How AI Is Increasing Systems Resiliency  Hi everybody. Thank you so much for joining us live today. Uh if you're in the US for ignoring your family uh on this big travel day or hopefully you're not driving. Uh and for those listening to the recording for our discussion around testing your system limits in the age of AI and in the face of so much more code than we ever planned and every other probability and variability that even people may not be involved in creating anymore and talking about if AI may or may not be increasing resiliency in this time. So this is a lead dev webinar created in partnership with harness. It's going to last about 45 minutes after which myself and the panelists are going to head over to the lead dev slack to answer more of your questions and I'll add that link in a minute to make sure you can join if you're joining us live. And it's just a webinar Q&A channel. It's a great community, the lead dev conversation around engineering leadership and how to support your engineers more, how to communicate better with the business side of everything because that's often the part of it that's the issue. And for those that are joining us live, feel free to share your questions as we go. I'm happy to reflect that and be your advocate in the conversation. So, let's get started. My name is Jennifer Rigggins. I'm a technology journalist. And today I'm joined by Helen, by Marcel, by Panos, and Martin. So, in that order, if that works for y'all, I'd love for you to introduce yourselves. So, thank you so much, Helen. >> Hello everyone. Uh, pleasure to be here. For those who haven't yet met, I'm the VP of engineering at Multiverse, uh the company building AI native engineering teams and enabling citizen developers to do things with yeah uh AI native powered tools to ship safely and fast. And with that, I'm going to hand it over to Marcel, I guess. >> Thanks, Alen. Hi, my name is Marcel. I'm head of platform at Dio. We're an observability startup um focused on open telemetry. Um yeah, a set of platform I'm in charge of our infrastructure. So things we discuss here are very uh yeah important to me, right? Building reliable infrastructure systems at scale. So I'm looking forward to the discussion. And next one would be >> Martinos. I can go. Hey. Hey everybody. Um I'm Panos. I'm the observability platform engineering director for Nike's global technology organization. Uh been in this role for three years. Um my teams own the observability platform for uh Nike's internal developers, operations engineers, accessories, you name it, we got it. Uh platform comprises of both vendor products as well as open source software and homegrown solutions. um multiple geographies um big scale fun stuff especially uh in these days around the year and uh Martin you want to go next >> sure I'm Martin RS I'm field CCR at harness harness is a uh everything after code software delivery platform so we work on everything that happens after code and and that our overall mission is to make that as easy and toilfree as possible And just to time stamp it for those watching the recording, this is Wednesday, November 26th. Tomorrow for those in the United States because Canada had it earlier in the month or last month, it's Thanksgiving. So Thanksgiving also translates to I think it's the biggest shopping week of the year. So thank you especially Panos for joining us during what we can assume it's one of your most stressful weeks of the year. >> Yeah, >> absolutely. Yeah, I know. It's a pleasure to be here. >> Thank you all so much. So, for those listening and joining us, the topics we're talking about today, feature flags, automated rollbacks, chaos engineering, they all come under come under this umbrella of what James Governor dubbed progressive delivery. There's a new book out of that name, progressive delivery book, I believe. And it's all about not releasing everything at once in order to decrease that blast radius. So when things go wrong, which if you're going for rapid releases, smaller pieces of code, rapid feedback, they will, but you're prepared for them. Or with charity majors calls, you're prepared for the unknown unknowns. You know more about your systems and you know how to respond. I recently interviewed Tyson Singer, uh, one of the leadership at Spotify, and he said his team of engineers, Spotify, roll back 42% of their feature releases. That sounds like a lot to me. I was like, oh, really? But then he explained that that's actually the sign of a mature team that can really test your system limits in production which can only happen if you have the right guard rails in place, the right security in place, AB blue green rollouts, roll backs in place using chaos engineering to understand you know what your systems limits are and this is where every or engineering organization every industry is different. So, we're learning today from four very different companies and at very different places in the progressive journey with their progressive delivery journey. And that's necessary in the face of AI because both AI is creating more code than ever, but it's also a definite part of this solution. It's not the only part of the solution. The human engineers are a big part that we really believe here at Lead Dev, and I think everyone on this call agrees to that. And there's a lot going on. So to start, I would like to ask everyone on the call, what percentage of experiments do you roll back at your org? And I'm also going to invite those that are joining us live to also vote and give your response as well. So Helen, do you want to go first? >> Yeah, for sure. I didn't have a firm percentage at hand. like it fluctates uh a little bit but I guess in general we embrace the idea that the roll back is just another step in learning. So like what we quantify in terms of like rollouts and roll backs is like the percentage of learning captured. So not every like experiment counts as a learning there are certain criteria that need to be met for this to count as learning. So we're less firm about like percentage of rollbacks but more like the ratio of experiments versus experiments that resulted in learning and that is around like 40% of all the experiments resulted in us actually capturing some insights from that. Um so yeah that's me >> and you said you have specific like numerical definition of what learning is. Can you explain more about that? Yeah, it's uh it's both like the qual and quan uh in that sense is like have we captured uh certain like novel insights or like have we captured certain gaps in our understanding of like either performance uh of the product or like some of the metrics sort of dipped be it conversion be it uh I guess like user sentiment if we have captured like enough novel insight Uh there is like a bit of a questionnaire. I'm not going to bore the audience with all the detail but like you have to check quite a few boxes to say that oh this was like a experiment that resulted in learning otherwise it just counts as like experiment without learning which like also they have to be a fair share of those as well. >> Fair enough. Martin, same question. And I guess both for Harness and previous companies. >> Uh, sure. I mean, I I would say at at Harness, we we do have the advantage of of really having that true experimentation thing. So, and and a little similar to what Helen was talking about really. So, yes, like when we're experimenting, we're experimenting with a single feature. So, it doesn't really it's not rolling back the whole thing. And often we're experimenting with multiple versions of the same thing. uh which is uh you know which which really drives that learning piece that Helen was talking about that kind of approach of saying hey I'm going to try it like this and like this and like this and I'm going to uh either maybe turn them all off and create a new version that's the consolidated learning or actually that one is just clearly the best and the customers love that so we'll go with that one and we will get rid of the other two and and just turn them off and shift all the traffic and I I think there's a nuance to that and you have to have all the right data to be able to do it. Helen was talking about that that you know it's there's a lot of questions to answer but I think as long as they're del the more often that they're deliberate uh I think the more value you get out of them for sure. I think previously uh I well I know it it was it was somewhere in that 10 15% um uh certainly at my previous or and that that was a lot of that was to do with uh you know just um not doing true experimentation as in hey we've turned something on and actually oh that was really a mistake. So it was more a roll back than an experimentation >> okay >> learning if that makes sense. It's more of like that was a bad decision. Let's just roll that back. That was justing rather than saying, "Hey, we think one of we know we're going in the right direction. We're just not sure what the best path is." >> Fair enough. Yeah. Marcel, what about your team? >> Yeah. So, for us, I would say maybe it's a little bit different since we're still a young startup, right? We are live since one year. So, the way we typically roll out brand new features is we have design customers first, right? And they have for them it's more it's more okay if something breaks right because you know that um yeah the feature is being developed on but um typically now they have the feature which is then alpha phase for a month or two um we onboard more design customers and then we go live with the feature and I haven't seen us disable or roll back um any major features yet Um, but since I'm an SR and also platform engineer, what I have to roll back every once in a while, and that's probably where the 10% would probably match pretty well, is if there are bugs, right, or if things are crashing or if there's errors introduced with new releases, with the new deployments. Um, and then I do roll back things in the back end, right, and disable certain things, new new code that was deployed. So, um, but that's then more on the infrastructure side. Not so much on the specific feature new feature side. >> Absolutely. And we can't forget I think platform engineering has reminded us more than anybody else. Our customers are internal at times too and probably be tougher if you're a platform engineer. >> Exactly. Right. And it's um I mean we use feature flex heavily also on the back end side infrastructure side. Right. And especially for especially for that use case that we want to be able to disable pretty much new things any time, right? I don't want need want to wait half an hour for new code to be built, right? I want to be able also on the as an S sur, right? That gets pinged at night to be able to bring the platform back to a stable state by disabling certain things that are causing issues, right? So that's why where we use feature flex for example also a lot. >> Okay. Fantastic. Very true. And there is a difference between automatic and a conscious roll back that you do as well that I'm sure we'll get into. Panos, what about you? >> I definitely don't have a number. Um, this is a heavily federated enterprise environment. People run their own things uh in their own respective domains. Um but uh what I will say is that we do other things as in not just feature flags or uh we have a very big let's say friends and family user group let me put it like that where um certain features that are like let's say heavy hitters um the latest one that I can think of was the genai powered search feature for the mobile apps you can we use feature flags to distribute access to this feature to control access to that specific group of people which extends far beyond like the scope of just technology right it's going to be like people from the business side etc etc uh and to echo some of the other uh panelists these these kind of longunning experiments will probably go on for months at at at our side yeah >> absolutely and I think something we should distinguish is next week on here on lead dev so you can join those listening. Uh we're talking McDonald's and JP Morgan Chase about how they're using feature flags and experimentation. And certainly it has to be different in different markets because Chase, a bank can't release a feature to a certain segment of the population. It goes against a lot of regulations. It can play around with buttons or colors and tests to try to increase signups or adoption of a certain feature, but you will know what you can offer or not. And just to let the the listeners know what the audience has voted, vast majority said 10 to 20% roll back. Two said they had no idea, which is totally fair. A lot of companies are newer on this journey. So, welcome. and one brave responded that 60% of more of experiments rolled back, but that could just mean they're very experimental. So, these are all great facts to bring with the conversation. So, thank you very much to everyone joining and of course it is very different for everybody and every different things. So with that in mind, we have this issue that Chaos Engineering was not the best named thing. Chaos Monkey, we we are grateful to Netflix to open source this and bring Chaos Engineering to the world, but the the visible concept of monkeys throwing their excrement at something, which is where the chaos of the monkey is. they throw their poo for being very prof in a professional setting is not the best conversation with board with board members. Even the word cha chaos is not something that people that don't own cats want to hear about at all. So there's this whole idea of chaos. It's a hard message to do. So how do you get started talking about this idea saying I want to experiment more? Especially in the time of AI where you're hearing this top down use AI but then but we don't want to express our data. We don't which AI is useless without internal data but how do we do it safely? How do you get this idea of guard rails and progressive delivery especially in a time where budgets are like AI focused? How do you make that connection with business leadership? How do you sell it? I I don't mind starting here. So I I often think that the conversation starts about resiliency and reliability. I mean that is often a good place to start that that conversation you know and even we will talk about you know giving something a resiliency score making sure it's up and that applies to everything right. So you know whether that's your infrastructure but also you know if you're if you are deploying models or agents or you know actually having the infrastructure behind them resilient you don't want your new fancy AI feature to be uh not reliable. Uh and often it's the thing that gets anything that's new tends to get hit really hard. Um and I think the other thing is is that the um there is that thing as well about just kind of sharing the impact of not doing it. So, you know, and most organizations can find a story of, hey, you remember when this happened and how much it cost us, uh, you know, actually including that in that resiliency score and saying, hey, we could do some small experiments that would actually help us be more prepared. So, next time the impact would be less because we'd be able to react better because we practiced, because we've done targeted experiments based on the way we work. >> Helen, you're nodding along. Yeah, that uh I can empathize with that framing a lot and uh like I wasn't super experimental when it comes to like framing this to the stakeholders. I was trying to kind of even though the the realities have shifted around us. I think uh the concepts around like revenue protection uh overall like resiliency, fire drills uh this is something familiar to the stakeholders and you can frame uh some of the like focused investments around those concepts because it still remains important for the business and uh in our case uh like we're also democratizing development with trying to bring like more uh citizen developers as we call them uh folk folks who are not like software engineers by trade. Uh we encourage that. So like we want to intentionally design for safe failure. Um right and when you're like empowering that broad set of people to build like guard rails are non-negotiable. So things like feature flags and kind of blast radius maps, but also fire drills, right? Like the good old fire drill is not going away anytime soon. and you want to make it interactive as an exercise. You want to sort of uh like uh throw a bit of drama and a bit of like murder mystery into it. And that's kind of the thing that we've been doing by like simulating certain very real life situations. something that happened to us in practice multiple times uh but just like having a slight twist at it be it like a AI generated pull request that passes some of the guard rails that like is invisibly dangerous in some shape or form or like could be like a bit of injection situation or could be like a new spin of on fishing that can take certain things down and just seeing like what the real life response is of people in the room, people in the office on that given day. And the biggest goal with that exercise is just trying to get them to orient quickly and figure out like, oh, when this and that happens, like what do I do? Who do I call? Like where is my big red button that I need to press. Uh so just making it more like organic as opposed to artificial. But yeah, just like injecting this little mischief uh on a weekly monthly basis into the culture. >> But when you say organic and fire drills, are these because we do live in London so they're very fire sensitive culture. We do have like I know on Tuesday the fire alarm it's going to bop at noon at my co-working. Are these scheduled regularly scheduled fire drills or are you just throwing them out there to see what happens if you throw a wrench in every day? >> They're regular, but uh there >> regularly scheduled. >> There is no pattern. They're they're scheduled in the background, but it's not like your typical Friday 11:00 a.m. Uh so like we're trying to mix it up. So you don't know what to expect. Like is that a fire drill? Is that real? And recently even the exact uh board has been through one. Uh so it's definitely like democratizing uh in all all shapes of forms both up and down, you know. >> Wow, >> that's pretty cool. Yeah. >> Yeah. Marca, what do you do? How do you get everybody on board with this up and down? >> Yeah. So, um what I've what I've learned in Yeah. last few years is we use these exercises. So we do table talk top exercises um like mostly for the on call people right and the backend engineers that are also on call and we do them quarterly and we do them what I've learned is that it it actually helps the team first of all with onboarding as well we are hiring a lot of people right and then they get to know the tooling right so we typically either take just take down random servers or even as we we have certain tooling in place right for on call and alerting. Um, and they integrate with Slack, right? So, we get pinged by incident. It opens the Slack channel. Then there's a Slack cuddle automatically, right? That people join, you pull in people. This whole process of where do I go like not so that there's not 50 threads in Slack somewhere all over the place, right? Where people say, "Hey, there's something wrong, but where they can uh can all join in one place, everything is documented there." And that's what I find really really useful in these tabletop exercises, right? um not just how to debug, how to use the tooling to find the issues and to see if the system automatically recovers with all the things that are built in or the resiliency, but also for the people that maybe joined new, where's the place I go, how how is the process, right? Who becomes the uh I mean the specific roles, right? That automatically that you can assign the incident commander and so on, right? like for the whole we use these exercises not just for the technically ensuring that the system is fine but also for onboarding new people into into that flow right and um that works really well because it's an it just helps people that have never been maybe in that role right to see how how we communicate also that it's blameless right that it's a safe space everybody can ask stupid questions right so um yeah That's how we've been using it. And um yeah, and then besides that, we have these um I would say they're probably required uh at least with our sock 2 audits, they were required that we do monthly disaster recovery drills for our critical data stores. >> If you're enjoying this, then lead dev's got loads more to offer and there's tons to explore. So the reason I love lead dev so much is they've got such a range of content on so many different topics in the world of engineering leadership and management. >> So lead helped a lot on building my own career, right? And it's great to also see how it's helping the new generation as well. >> Make sure to sign up and join us at leadv.com. And they yeah they are a bit annoying because they happen every month and so far they've always been fine, right? And but it's just mandatory that we do them. So yeah, but again, yeah, the the tabletop exercises are actually more fun because more people are involved. Um so and it's not as stressful, right? Because uh you know it's an exercise. stuff >> and whether it's an exercise or you don't know what you said about the blameless culture has to be there or you will >> Yeah. Right. You need to take the pressure off people, right? And it's better to do that already in the exercise so everybody sees how to communicate, right? Because if something then actually happens and you're I don't know at 10 p.m. or 1:00 a.m. sitting together in a call, uh it's good to know already the drill, right? Um and yeah that open communication is welcome right and no stupid questions are >> that there are no stupid questions right >> you slightly on that in terms of like um having that disclaimer or like it's just an exercise as opposed to the real thing I mean it's not my thing to make people paranoid about everything and make them suspect every single thing like oh is this real is this actually happening but I think the response under pressure is quite different. >> It's different, right? It's absolutely different when it's it's a real thing and you don't immediately know what's what's broken, right? Uh yeah. So, we don't tell people what's broken also on the exercise, right? They just see alerts popping up, but um they know it's an exercise, right? So, there's still this a little edge that's missing compared to the real thing. Yeah. But learning flow of communication and how to just practice talking to people when you are well rested helps when you are inevitably not well rested when you're waking someone up. >> Fair. >> I mean, there's a lot of communication you need to do, right? If something breaks, there's a status page, right? You need somebody that t takes care of that, right? Then there's somebody that needs to drive the screen, right? During these exercises and all that, right? So, and that needs to be practiced, right? to so people know their roles um when the real thing happens. >> So important. Oh, Panos, being that it is Black Friday month, I guess it's called now. Uh it's been going on and on. Are you on call right now? First of all, >> uh I I think we're always L3 anyway, but uh no. Um I I don't respond to incidents uh unless it gets uh pretty ugly and uh that hasn't happened yet. >> Fingers crossed. >> Yeah. Yeah. Yeah. So I feel like there's two different questions here, right? One is um well chaos engineering sure Nike. We'll talk about that. Maybe the other one is like how are you measuring resilience, right? Um all right, let me let me start by um chaos engineering, right? Um there is a separate team. There's an actual chaos engineering team within our resilience engineering domain. It's about 3 years old. The practices themselves largely precede the creation of this team. Um because the global technology organization is the size that is about 4,000 people. As you can imagine, this is a matrix organization, right? Comprised of multiple federated domains. Uh so we are mostly talking about an opt-in intake process. So you want to do chaos engineering for your services as a service owner. This is like your playbook right there. Um that said, we also target certain teams and services uh because of uh consumerf facing features or experiences that are about to be released. Um or identified risk to holiday like we have right now or potentially past incidents if a service is prone to not meeting its SLAs. Um we have a a formal high availability disaster recovery uh program um that our minimum viable consumer journey services think anything from being able to register on nike.com all the way to successfully checking out your order. All these services are going through this uh program on an annual basis in preparation of the holiday period that we're on right now and chaos experiments are very much a big part of that like um to give a kind of a glimpse into that we literally kill um an AWS region uh during a weekend let's say a few weeks prior to today and do the whole thing uh live on on the live environment at that scale. the skills about um 300 microservices for the MVCJ scope. Okay, so that's uh what we do with chaos engineering. um how we measure resilience is a much more like we don't have enough time but long story short the holiday program that I mentioned um it has a crew right those set of people um maintain some visualizations think dashboards that uh attempt to quantify resilience KPIs at scale um one of them is a graph of MVCJ services with their respective SLAs over aggregated over a time window uh as well as uh another visualization that we call a resilience scorecard uh that tracks 10 different dimensions of a systems resilience posture. Uh think observability standards adherence um chaos engineering practices uh performance metrics for consumerf facing APIs for example etc. >> Wow. And everyone afterwards going to be like Nike. I just think it's funny because what could be more built for brick and mortar than the idea of Black Friday when a lot of people are off on a Friday that we make it the biggest shopping day of the year. And they tried to do Cyber Monday, but we know that never worked and that's why it's like a whole month now. But if you think about it, it was built very deeply analog and now you have to fit it in a very deeply online world. So, good luck to y'all and I think you're doing well. Now, we can't have a panel in 2025 without talking about AI. So, we've talked about how it is impacting everything, but AI can also help in the daily site reliability engineering work and things. Marcel, you talked about how I actually found you because I saw you speaking at SRCON about how you use AI in your daily work. So, do you want to kick us off on that? >> Sure. Um so we've experiment actually our our CEO enforced us upon us that we should try out different tools right uh starting end of last year I would say um also just to learn about them right to understand how they tick so we could see um if they are valuable for us right and so we tried out different tools and at the moment but these things are changing right at the moment we ended up with cloud code for example for adding new code on the programming side right that's mostly Golang Java NodeJS they use it heavily but also on the infrastructure side right we have a lot of go we have a lot of terraform and other stuff it would just works really well for me um daytoday and um so yeah that's that's one part right then there's some of these uh yeah another example I had was you know there's as a manager or as a yeah or a team lead whatever right there's some tasks you need to do maybe weekly that you you don't want to spend too much time on right so reporting right what is the team doing what's the status quo we use linear for all of that to for the ticketing and it has an MCP server so since I don't know six seven months I've been using simply claude and that MC MCP server integration to tell to write me a nice summary right like a management summary for the weekly reporting what did my team do right what did it finish what is it working on next so I don't have to write this anymore and it's really like these save me some precious yeah minutes or hours right every week uh so that's nice and yeah then in our product as well right we um build our own agent for analyzing all the observability data for debugging and I just use that regularly right I I'm on call I get errors I just Our agent is called agent zero in the product. I just call it it digs through logs, metrics, traces, writes the prom queries faster than I can. And so this is my experience, right? So these are the different levels of where I use it and I don't have a number how much quicker it does or how how much it helps me or how much faster it does me quicker it makes me but uh it just takes away some some cumbersome uh work items I had to do before manually right or myself. So I I very much appreciate uh the development things that are happening there. >> And I like that you use the example of something that was less technical but helps improve your daily life. >> Yeah. There's always these annoying toil or chores, right? Exactly. >> That you might not want to do so much. Yeah. >> Yeah. That's exactly my mantra, Marcel. I guess uh like along the lines that you were saying, there's quite a bit of tools that like are willing to take away uh like the creative aspects, but I'm like take my maintenance. And that's a bit of like my pet peeve. Um and one of the biggest levers that we've picked up from this uh kind of wave is being able to do fleet patching. uh the concept of like oh if like another uh vulnerability is discovered in npm package I'm able to patch it like across multiple different repos in hours as opposed to days. Uh, and it's just like it's such a nice thing to be able to have in your like at your fingertips to be able to roll out hundreds of uh, pull requests in a matter of hours and have them merged as well as opposed to like just the cost of coordination of like going around and aligning making sure that like every team knows that they need to respond to this. You can just send PRs, right? >> And these npm vulnerabilities seem to be increasing. I have the feeling right. >> Exactly. arrive again. So, >> exactly. Very fresh and just like touched it yesterday. >> And and some are AI driven. So, there's a dark side to this as well. >> Yes. Exactly. Right. >> Yeah. Yeah. >> The box is open, right? So, it's not it's already there and you kind of have to know embrace it to fight it kind of, right? >> Yeah. There was there was one recently where the AI was actually rewriting the virus itself along the way. So refactoring the code which I think Google found it uh in doing some experimentation but blows my mind. >> Concerning but terrifying even since I just saw Megan the Megan with the three in the in the title fictional horror film. It's quite >> but it's also we're getting closer to reality. So we do have to recognize that AI just makes everything more exponential in growth and the vectors as as well. So something you mentioned before uh Helen is that you used AI to help you with generating these fire drills because I think the fire drill is a very interesting exercise probably sometimes for your staff but for definitely the people that want to ensure res resiliency it's really good. Yeah, like there are quite a few like creative things that you can automate from that perspective and creating uh challenges is like one of those things as well. Like you can feed it like multiple different uh criterias as to like it needs to look a bit more normal. It needs to pass x y and z and then you discover very surprising things that you didn't like. It's almost like red teaming. uh but doing this with like an agent being your partner in a way uh so that definitely not something that I I thought I would be doing but it's quite accessible now that you can actually do it without being a security expert and it produces the amount of learnings that is surprisingly good >> absolutely that's a great what about you panos how do you use AI in this or do you decide not to maybe during during your no code days. >> Uh I'm just envious of Marcel's quality of life. Um like Lord save me from the AI tooling sprawl. Um I think that's something that we discuss a lot internally, right? Um certain realities are a constant in enterprise environments. Um Microsoft for example. Uh and then there is this whole new breed of tools um that are amazing. Claude was mentioned which is absolutely brilliant. Um but unfortunately the effectiveness of these tools is directly proportional to the data they have access to and uh especially in this kind of federated uh context. So um we do use pretty much everything under the sun. Uh so for example we have cloud, we have cursor, we have uh copilot etc. And we do pretty cool things with um agents in this space try to shift the conversation left as much as possible but um honestly your mileage will vary depending on the specific domain that you are trying to uh um uh change. Uh a good example from the retail industry would be supply chain. uh all systems related to supply chain are proprietary. Um the effectiveness of AI systems in completely novel code bases is um unfortunately not up to what you would otherwise expect. Um but uh I think the main problem we're trying to solve right now is consolidating um not so much the tooling but the data. um how can we provide uh consistent access and definitions to large language models so that they are simply u more useful to people. >> That's very cool. Sorry, I I was just listening to that and and there's a I mean there's a couple things that that just stood out to me. The the one the one thing about like having a single interface into as as many of those tools as possible is something that that you know even as we've been building our own tools we've you know we would very much say the same thing you know we would say we have for example in our case we would say we have harness AI even though that is multiple agents under the cover because we don't want people to say I have to go to this thing I have to go to that thing I I just want to use the AI and have it know the right place to go to. And that's the same, you know, whether you're using Claude or anything else, right? You you want to link those things together. And the other one that was super interesting is that thing about, you know, the the how useful they are is how how tied to the information that they have and the context the context it has to be able to be useful and meaningful. So I I which I kind of love, right? Because you know I like as an example if you're uh building out chaos experiments for an example is quite time can be quite timeconuming and can be quite laborious and actually AI could be useful but it's only useful if it understands the topology of your you know your application your architecture and some of your flows and then it can do intelligent things and say hey now because I know how you deploy and I know how your network flows. Now I can say here's some really good experiments for you versus just a here's a set of generic experiments you know so that thing about you know having the right data and the right access is really key to having good AI you know and that's something I hear from our customers but it's you know personal experience as well it's even as I'm using it internally it's it's definitely the way the way it works better the more data and and context it has the better it is >> absolutely And I quote literally all the time in every article at this point. The data I've been looking for from MIT that did a four I think it studied AI across four billion dollars of investment and 95% of AI pilots fail and it's because the data silos. So that you need to work out with the guardrails. But I appreciate that no one said we are going all in on Microsoft or whatever the whatever the thing is whatever the brand is the mark with the AI because it's all so new and they keep edging each other out and that's kind of the nature of the beasts or beasts. So I think what it comes down to is you can't keep your developers from using AI, but you got to have the guard rails in place and the S sur the site reliability engineering and resiliency engineering practices in place to actually use it because they're going to use it anyway. And you really need your developers to do that and >> you need them to be safer. So you need to be safe to fail as we said and you need to be have a blameless culture. So I want to say thank you so much to all four of you panos Martin Helen and Marcel it was really wonderful to learn from you. We're going to continue in the Slack because we just had a question. So we'll definitely continue that and those that uh to entice you entice you to also come to the Slack community. The question was after activating AI, MCP and similar tools in the platform observability and on call platforms was there any improvement? Which metrics do you do use to track here? So I know this group I know we can spend the next 15 minutes in Slack or the rest of our lives talking about metrics. So I really look forward to that conversation considering continuing uh our Slack is a great community. So do join that and if you want to play around with feature management on December 17th harness is offering a free workshop on from future flags to measure impact which includes a way to measure and take home your experimentation maturity which I think you may think you're exper mature in your experimentation or not but metrics are always the most important thing in bridging that divide between engineering and uh business and then on December 10th You can join me along with one of the authors of the new progressive delivery book and friend of lead dev Heidi Waterhouse, Coral Logix's Annie Freeman and Wisees another deeply regulated company Rick Kle to talk about the guard rails you got to have in place before your ai program crashes and burns. And I really just want to thank you all so much and I hope we'll talk soon and well we'll continue the conversation in Slack and good luck with your experimentations and whatever this season brings. Happy New Year., How AI Is Increasing Systems Resiliency, YouTube, 01kb682ttb2zg5zvgtzms55ncb.