Derisking Releases in the AI Era  Hello everyone and I just huge thank you to everyone for joining us today for today's panel. We're talking about d-risking releases at the speed and impact of AI. This is a lead dev online panel discussion created in partnership with harness. Discussion is going to last about 45 minutes after which both myself and the panelists will head over to the lead dev slack. uh if you have any questions you can ask throughout here and then if they continue or if you want to continue the conversation that's a wonderful place to do is the hashtag panel Q&A. Uh for those joining us live, I'm going to share in the chat right now the link to that lead dev and I will add as a resource because it's not ephemeral and or it is ephemeral on Zoom, but it's a really great community and I do recommend joining it. I like a Slack community. I don't know how others. So yeah. Uh so we'll answer questions if they see fit throughout. Otherwise, like I said, we'll continue on there. So you have the Q&A tool. Please use it. So, let's get started with some introductions. I'm Jennifer Rigggins. I'm a technology journalist. You may have seen me host a lead dev panel before. And today, I'll be joined by three lovely panelists that I invite each of you to introduce yourselves. Claire, you are at the top of my screen, so I let you go first. >> Thank you very much, Jennifer. Um, longtime lead dev fan right from the early days. So, lovely to be on this panel. Um, currently director of engineering at NAN, heading up all of the enterprise side of uh, engineering there. Um, spent a lot of years working in tech um, across the globe um, but based in the UK um, and some of you may have seen me more recently on panels uh, representing Netlefi. Um, recently joined NA10 for a new scaleup challenge. Um, I'll maybe ping that over to Liz now. Hi, I'm Liz. I'm a technical fellow at Honeycomb and I previously worked for Google as an Nesserie for something like 11 years. Um, so I've been around this block of uh tension between release velocity and reliability for years and years and years and I'm very excited for what the future holds. Ryan >> and hi everyone. My name is Ryan Ryan VA. I'm a feature management and experimentation advisor advisory director here at harness. Uh I have the opportunity to work across a broad broad spectrum of uh organizations who are implementing uh feature level observability within their groups and I'm excited to to share you know my experiences working with them with you. Okay, just a little background as people are jumping on. They came to hear from y'all. But at least in open source, the use of generative AI directly drives a three to five multitude increase in lines added in the first adoption month in projects. This gains dissipate after two months. It's not a bad thing. uh lines of code not usually a great metric for anything but I think it's actually a pretty reliable predictor of complexity. So while we don't want to measure lines of code in individual level for various job safety and security and psychological safety reasons it's not bad one to measure of your complexity of your database. So also it's been observed that persistent technical debt accumulation static analysis warnings have increased by 30% and code complexity increases by 41% post adoption of AI. The 2025 Door report, which I'm a journalist, I love these percentages and things, said that AI tends to increase pull request size by 154%, which of course is increasing dramatically that blast radius of any release. And I think Doris said it very well that it called AI an amplifier and a multiplier. So honestly, I don't think AI changes the things because if you were doing the DevOps and engineering best practices and guardrails that we've been talking about for years, AI, you should be logically able to keep up with this new pace. You probably also aren't releasing as much. You are not adding that many lines of code because your quality checks are in place and things. But it is if you don't have those in place and you don't have these progressive delivery best practices that we're going to talk about in place. Well, every release runs a certain risk, but the larger release and especially those written without the checks of human beings in the loop run the risk of more damage to your reliability, security, reputation, bottom line. So, we're on the call. Spoiler alert. We're all big fans of this idea of breaking work into smaller manageable chunks reduces risk. It's not an original idea, but this idea of then progressive delivery practice that can reduce that blast radius more. And these are just so much more important in the face of managing deployments with unpredictable AI elements are involved. So yeah, that's our topic. That's what we're talking about today. So we can help you figure out how to have the smaller, more manageable releases and the smaller man more manageable code and impact. And I know from the questions ahead of time, there was specifically people looking for how to do that under this idea of enterprise breath and complexity of code. So without further ado, I like a definition. to start off. Claire, would you like to define what you think progressive delivery is? >> I think there's people that have defined it better than I can, but I can certainly describe what I would expect as as to happen. Um, so for me, progressive delivery is um rolling out smaller feature sets to a subset of the audience. Uh so you can constrain it in two different ways. uh as in who who is affected and how much of the new thing you're pushing out there and whether you're running things in parallel commonly done with with feature flags and uh canary rollouts and things like that but I don't necessarily want to uh dig into to specifics like that right now I think the bigger the bigger conversations more about how we wrangle all of this >> and then something you did mention about progressive delivery that I thought was very interesting that there's a correlation between companies companies that do have the practice of progressive delivery and like positive work environment. >> Yeah. So I've I was uh a big fan of feature flags from when I first encountered them at GitHub um a long time ago now, many many years ago. Um, but I have definitely then seen other companies that they're just not even aware of them or why you want to use them or having the maturity to to design um u changes that can take advantage of them. Um, I think that this is to the point you're making that this ties to me the trrimodal nature of tech companies um where you've always got people who are pushing the bleeding edge. um whether that's AI or anything else, but it's adopting the good practices. It's doing the DevOps is doing all of the things we've been talking about for a while now. Then you've got the kind of leaders within countries that are not necessarily completely there but are doing their best and trying. And then you've, you know, some people on cloud migration still okay, you know, how long have we had cloud? They're the kind of third tier of ones and they're the ones that even now I would suspect don't have the the organizational capability or even the infrastructure capability to do progressive rollouts. This is like back when I started with gold CDs being pressed for your master version of the software that you released. Just dating myself slightly >> Ryan. Yeah. What do you want to add to that on the psychological safety? Oh yeah. Yeah. You know, um, you know, progressive delivery is a is a is a very interesting topic to me. I've been preaching it for a very long time. Um, specifically around using obviously, you know, feature flags to kind of support that type behavior, although there's other mechanisms to to do that as well. But, um, I was just thinking about psychological safety. I was thinking of a conversation that I had um, with a VP of engineering at one of the largest credit bureaus, you know, in the world. and they were talking about kind of their efforts around application modernization which is also kind of a buzz word and you hear a lot of people talking about it. Um they were very interested in kind of helping their engineering workforce modernize the tools that they were using. You know they were using kind of legacy application legacy languages. They wanted to to take advantage of you know more modern architectures. uh and they thought as an organization that if they did this did this modernization they would see an acceleration then in um new capabilities coming to the market right just a higher velocity rate. What they discovered was that they did modernize their tech stack and they were using more modern architectures. uh unfortunately it was having the unforeseen consequence that a lot of the testing was shifting you know is they were shifting to more testing uh within development because they didn't have as much experience working with these modern technologies and stacks and so they felt like they had to compensate with testing more and so actually things slow down as they as they kind of ramped and adopted these new technologies. It wasn't until they started using feature flags, they kind of created that safety net that they then saw an acceleration because psychologically for the engineers there was less risk in that release process. So great that they're you know now serverless or you know using lambdas or micros service or microrends and all those sorts of things but ultimately the developer feels the most risk at that point of release and de-risking that was really kind of what unlocked it for them in terms of you know feeling safe to to get ideas out faster. All systems are sociotechnical systems, not just the technical system, right? Like no technological change is actually going to make a difference if your team is afraid to interact with production in any way because they, you know, it pages them all the time or they get yelled at every time they cause an outage. >> So we have feature flags. Maybe we want to quick to find that in like a few words and then let's try to run popcorn. >> I think we should back up a step though, right? like you know when CLA was talking earlier about this idea of um rolling out features to a percentage of users right like I think even going beyond a step beyond that the critical insight that progressive delivery had was that you can decouple a release of software uh from a deployment of software right um so when we are enabling the ability to turn on specific functions of our code at times of our choosing rather than at times are not choosing. I think that is the key thing that then enables teams to better manage their own releases, right? Like and then we can talk about the details of feature flags from there. >> And then when we say roll out, for those that aren't as familiar with this concept which James Governor kind of coined as progressive delivery, how do you decide who to release to or not? Certainly people um do everything from, you know, hey, we're going to roll out just to 1% of users at random. There are also circumstances where you might have early access customers, right? Like you might have customers that have opted in. Um we actually just completed a pretty long process to um turn on a new metrics back end at Honeycomb. And we had customers that wanted to opt in early. And then we had customers that were like, you know, I don't want to be broken, right? Like, you know, please don't please don't give this to me early, right? Like, and that self- selection was really, really important for uh people to get that um early access if they were more risk tolerant um and they wanted to influence our future development and for pe for us to not break people that wanted to be a little bit more conservative. Um, and then, you know, there are also the people who decide, you know, these users are free tier. I'm just going to experiment on them, right? Like they're not paying me any money, which is a fair business choice potentially. Um, but I I like thinking first about, you know, who's more risk tolerant and then kind of figuring out, you know, how you divide your users into risk tolerant and not risk tolerant from there. There can also be building on that um uh customers that maybe on paper are not risk tolerant in general, but they've had an issue with something and you're trying to roll out something that helps solve their problem. Um and most of the time in most of the businesses where I've had direct communication with the customer and these things, they're like, "Yeah, sure. We'll try it out. It might solve our problem. We appreciate you taking us seriously." And it is actually helpful to build relationships um that way, which is to me an extra bonus on then making sure that the person who's got the real world problem gets the fix properly tested as well. >> Yeah. And that always jumps out at me is that that that's really well said, right? And that relationship building because to me the release process can be a great learning process. And if you do it progressively, if you do it incrementally, if you target how you release things, it provides opportunities for conversation, right? It provides an easy opportunity to say, is this working? Does it do the thing that you wanted it to do? Is it conflict with anything that you previously were doing? Um, and you really get to enhance your ability to align, you know, on the outcome that you're delivering with your customer, right? You can really, you can do some course correction, right? If you're not just turning it on for everyone and, you know, getting some feedback and figuring out, oh well, maybe there's an iteration on this that we need to do before we kind of go a little bit more broad around it. And so to me, progressive is, you know, there's the incrementing component of progressive, but it's really what can I learn at those increments and can I learn something different? Like one might be functional, another moment in that progressive delivery might be more around scale and reliability, right? As I increase >> and performance, right? Like you know, if you see 10% CPU increase from turning it on for 1% of your users, you go, hm, okay, right? like that needs a little bit more work. >> Exactly. So, you know, you can learn different things along the way and that's what opens up your mind in terms of deciding, you know, do I go 1% or 10% or do I go, you know, a targeted list of alpha beta customers and then into a percentage or maybe I never do a percentage, but I'm, you know, I I work with my community to get feedback at every step along the way. >> Right. Exactly. Self- opting into feature flags is also really really awesome, right? like you know every time you have an experimental or beta slider inside of your product right like that's something that you should also be managing through your feature flag framework but I think to Jen's original point about uh progressive delivery and about splitting up changes into smaller increments right like when we're talking about splitting up into smaller increments we are not just referring to the code right like I think that is the critical thing right like we are trying to get feedback faster accelerate our learning cycle not just by shipping code faster and and in smaller increments but also um you know microlearning from smaller numbers of users faster rather than you know waiting to deploy it to everyone before we realize oh we built the wrong thing. This is very interesting because usually the example for progressive delivery is roll out to well Australia or or >> hey stop picking on Australia. We may be very few people and we may be in a strange time zone but like you know we're real users. >> Yeah. And also the idea is like you're not being that responsive if it's not when your most of your staff is working. If you're zero or another Aussie company and you want to release to Australia first that's that's a choice maybe logical but I think these ideas of thinking about it more strategically as you said Ryan to have a conversation is better but on the other hand you can dump it on people that are not worthy. uh back when James Governor I feel like it was 2017 but I don't know first was introducing this concept or the concept existed but like coalesing around this term progressive delivery he said like one client he knew did dumped everything first on the people that used illegal or stolen credit or fraudulent credit cards as the first to me that that's lower risk and I get that and if you're going to risk they're not the customers you're really worried about. I guess on the other hand, it doesn't really show to me that you care about what you release that much and you want that feedback. It's really just is this going to work or not? Not this is a beautiful feature we are bequeething to others. >> Honestly, I think more teams where possible should dog food, right? like you know right like Honeycomb's deployment cycle we automatically deploy whatever is at at main once an hour >> but it goes through phases of first every Honeycomb employee gets it for our internal uh instance of honeycomb before it goes out to production right so that also is a a form of progressive delivery right like you know yes it's progressing through environments but like they're not throwaway environments they're real environments that our employees use and we will scream if we notice something wrong. >> That's the platform engineering way. You're the worst and the best customers. Best tightest feedback, Luke, Claire. >> Yeah, we we really are the best and the worst customers as developers. I mean, we we did that at at GitHub. There was a staff bar and things were enabled there. Plus, to Liz's earlier point, we we had it was possible for customers to opt into certain feature flags themselves when they chose to as well. Um, again, at Notify now at NA10, we're huge users of our own product. uh even if what it takes to build the product isn't exactly what you'd necessarily use uh for the actual say software development life cycle. Um but we we use it for all sorts of things internally because then and we have what we call the internal instance which um to Liz's point yeah that does spot a lot of things before they hit the wider audiences. I think one interesting AI tiein here is that we've talked a lot about how AI puts stress on the system, >> but I also see AI as an opportunity to make it easier to do progressive delivery in a lot of ways, you know, >> right? Like development teams struggle when you tell them, you know, hey, can you write like, you know, can you put a feature flag around this? And they're like, I don't know how to do this, right? Like >> write a skill, have the agent create the feature flag, have the agent wrap the code. Done, right? Like it's not just a technical debt generation machine. It can also be a technical debt paying down machine. >> Absolutely. >> Yeah. And and you know AI is also going to support that debt elimination, right? The idea of understanding what's in use inside your application as it relates to a future flag. Is it being called? Is it routing traffic? Is it no longer being modified? You know, and AI, you know, we're investigating ways that AI can interrogate your flags and say, are you worthy of still existing? Right? And if not, we're going to we're going to do a pull request uh and we're going to suggest that you get removed. Right? So all all stages of the life cycle can be affected by AI as it relates to flag management in our opinion. >> Also, what about other things? We're talking about feature flags, but there's so much more to progressive delivery. What about has anyone used blue green or AB testing where you aren't really sure where you want to go and which is the best feature to solve this problem. How are you going about that? I think that with regards to blue green for infrastructure like I think that that is a good thing to do if you can afford to do it. Um I think a lot of the challenge of blue green is that it does require doubling your resource footprint which may not be palatable at internet scale. Um however infrastructure as code is making it cheaper to do that because you can certainly you know temporarily provision that uh that blue green environment uh during the uh overlap period and then decommission the old one whenever you're satisfied. Whereas previously it was a okay now we have to keep a blue and a green running permanently. That's not so great. Um, >> oh yeah. >> I I think with regards to the other thing that you suggested, I I think, you know, we definitely can I I I think I think it's definitely worth worth doing. I I Yeah. >> Yeah. Okay. >> It it it it is um I think experimentation was an odd word to bring up with product engineering groups 10 years ago. Um, I think more and more I see engineering leaders adopting it as a key way to, you know, quantitatively measure the impact of of what they're doing. Uh, it's not a replacement for qualitative engagement. Um, but it can enhance your overall again that learning experience. And I think AB testing I see it more and more being adopted as a pattern within engineering to you know answer answer questions and and answer them quickly. You know if we tie this back to AI um you know in AI development I always say that there's a lot of knobs and levers in AI. You know there's um things like uh models and there's temperatures and parameters that you want to adjust. there's prompts that you feed into it. Um, nobody seems to get those knobs and levers right the first time. And so the ability to quickly um change any of those individual components and then have a feedback loop in terms of what's the outcomes then from that adjustment. Am I seeing less hallucination? am I seeing better, you know, outputs from, let's say, if I'm doing a Gen AI, better outputs. That's very informative. And so I think in the in the AI world, it's underlining the need to do AB testing, experimentation quickly. Um, >> and in fact, the AI world has a term for this, right? Like it's evs, right? Like you want to constantly doing EVELs, right? Like you know, no one disagrees with that. I here's the risk that I for that I do foresee with regard to AB testing, right? Yes, it's good when it's invisible to the user. >> I worry that with the pace of AI development, the bottleneck becomes the rate at which you can churn user interface and right like users do not like it when you move their cheese and being told you're in an AB experiment test group, right? Like you know, and we moved their cheese to see what how you reacted to it, right? Like at some point it grates on the user. Um, so I think we're people are going to have to figure out how to prioritize, right? Like doing these AB tests and not testing every feature on, you know, the same set of unsuspecting users because they will get very unhappy when you keep turnurning their UI. >> Yeah, there's an interesting tension there for me with the AI side because you've got the the constantly changing churn and yes, I've had enterprise customers at previous jobs say, "Why is the interface changing?" And you're like, "It's only changed once in a year." And they're like, "No, not happy." Um, but we're also like there's the hyper localized hyperpersonal software that's being built at speed and like where does this all be play out over time. I think I don't have answers but I think it's fascinating to live in it and see what's going on to be honest. Um where I would also say there something we haven't mentioned here where AI can help with the experimentation is I've got some of that happening where we are now at NA and a couple of my engineers have actually used the AI around the analysis and around what to try next and what to tweak and and kind of use it as a investigative tool um rather than a hit let's just build a new UI or build or whatever and I we've found out some interesting facts that way that we might not have stumbled across otherwise which is an interesting angle. Can you give an example of like what you found that way or how that worked? >> Um, we were trying to because obviously when you run experiments, you kind of if you're doing it right, you're trying to figure out what is the is this good or bad? did we move a needle? The needle wasn't moving in the way we expected. And instead of just being able to shrug our shoulders, it enabled the team to do much more of an in-depth investigation into that and then come up with some possible causes based on that which they are now running further experiments on. So I don't have uh complete conclusions there. But it it opened up roots that we might not have thought about taking or like prompted us to think about it more deeply. and the engineers took this on themselves and were like, "No, I need to understand this better and and just really hammered it and I I loved that." >> One of the areas around progressive delivery, we're talking about like we've been talking about lowering and limiting blast radius. The other side of that is because of the rapid feedback and that's something that both AI can make better. It's known for that because there's the prototyping, there's different options, but also if your PR site's getting longer or the bottleneck was never writing code. So if as it moves down the line, there's a slower feedback that way or maybe even this idea that developers didn't create the code, so they're waiting for feedback and they don't fully understand. So, what are ways that we can tighten this feedback cycle with faster releases when we're releasing so much more? >> I I think that um I think you have to lean on quantitative measurement a bit um to get a faster feedback loop. Um I think picking up the phone is still the fastest, right? engaging with your customers closely on individual changes. But unfortunately, that's also kind of a little more anecdotal in nature and you'd be hesitant to take strong action based on a couple of data points. Whereas, if you can measure things quantitatively, you should have greater confidence in behavior or action as a subsequent, you know, you know, subsequently from the signal that's delivered, right? I saw things improve or I saw things degrade or I didn't really see any impact on significant metrics. And so um I think quantitative measurement enables the fastest operational process. Um immediate feedback though you you can still attain you know qualitatively. I think there is one interesting challenge which is we know that language models do best when they get immediate feedback right like you know when they get immediate feedback on your tests are not passing right and how do we connect that loop back with production with what users are seeing for reliability signals right like you know we should be connecting them to our observability tooling albeit you know that might be on a delay as far as you know how often you release right like you know it might take half an hour or an hour you know or heaven forbid you know 8 hours for that for that to go out right like and then for you to be able to feed that signal back into language model of this is how it's actually performing and then I think you know things like amplitude mix panel right like things that are for user analytics are another place where you might want to feed that into language model although that starts getting in the domain of user requirements which I think is more of a human consideration rather than a uh thing that a language model can directly do something about >> yeah I think the the struggle that you'll have on the observability side is that you know it tends to not be cognizant of maybe individual changes that might be occurring inside the application or you know you've got a situation where of course many things are constantly changing inside the application whether it's you know other other new features being introduced or new capabilities or new infrastructure or new networking or new security what have you. there's a whole multitude of things and so observability tries to take all that in and tries to find the problem you know that that's in the noise uh and you know I think having a better sense of feature level observability so changing the paradigm from measuring the application to measuring the individual change uh is important to you know isolate that signal faster to take that action Okay, something we did talk about was this sense of ownership in the age of AI. So even if AI is involved in or wholly creating things, who is in charge of the delivery of that thing and who owns it? >> Definitely been a big one for me. Um I'm very strongly of the belief is you build it, not necessarily write the code, but you build it, you kind of own it type of um behaviors. Uh I know there's various or or structures that tweak that, but by that I mean I don't mind which tools you use. If you want to tap it out in binary by making a noise on your desk and have the sound recorder convert that into code, you do. You go ahead. You'll be very slow. It won't be great. If you want to use an IDE, great. If you want to use AI to help you, great. But fundamentally, you're the one that is shipping that. There's still the human element of that. So, you have to own that. To what level you want to understand that is probably down to the individual's confidence level on a bunch of things. Um, but again, it's like at the moment, we can't have agents sign legal contracts. We can't have can't page an agent at 3:00 a.m. I know there's a few firms trying to claim that we can do that. Um, but we still need humans for sure in so much stuff. that this is >> and also more importantly who is doing the learning who is feeding that back into architectural strategy right like >> there is a crucial role for humans to play sure we might be moving a level up of the abstraction stack but you cannot remove humans entirely or you just washing accountability >> yeah yeah um I've actually got a blog post half written exactly covering the kind of the learning angle of that because yeah um the experiential stuff might be going away and we need to find ways to educate people But yeah, I think the ownership does still very much sit with the humans, be that engineers or people like me as a leader or whatever. But yeah, >> so now that we've established who should be the buck stops with who is running this experiment even if AI is implementing it. So how do you safely experiment in this time period? What is necessary for that? I think that um I think safe experimentation um is an understanding of the audience. I think Liz kind of pointed it out well. It's like we're all human, right? When you run an experiment, you're interacting with people and you have to take that into consideration. Um I think a a safe experimentation program uses some trusted methods, right? True randomization. Um it could potentially do isolation of experiments, you know, keep out, you know, control groups, you know, from being exposed and using that as a baseline constantly to to make sure that you know that um you are preserving um you know your users access into the system and and and testing in a you know a responsible way. But I I I don't know that anything in AI changes that shift from an application development perspective. That's my my thought process anyways is that AI is it's software at the end of the day. It has end users at the end of the day. We have to be responsible to our end users. Um we have to be logical in what we're testing and you know are we creating an opportunity to learn something given the parameters of the test and you know what our hypothesis might be. Uh and if you and if you're confident you know you can say yes then you've got a safe test on your hands. I think much like the ownership angle there the the person working in growth be that a PM or a growth person or even the engineer they can't abdicate the responsibility around how you would design and then roll out the experiment much like they've done in the past um that still holds >> yeah I would say responsible experimentation requires that you have a well functioning release process. It requires that you have feature flags. It requires you have observability, right? Like there is a certain level of operational table stakes maturity that is required because if you are just flipping flags, right? Like and you can't actually measure the results like you are not actually experimenting. You're just injecting chaos, right? There's a reason we talk about chaos engineering, right? And I think that when we talk about progressive delivery, right? Like you have to actually deliver something and validate it. H >> it's like having a light switch in this room controlling a light in some other room and you're just flipping it on and off and you have no idea you're causing chaos. >> Yeah, absolutely. Uh we did have a question from the audience. So there it's a very broad one. So the idea of how do you quickly and safely recover from incidents after a release when many subsequent releases or deployments have happened since. So this is definitely a smaller releases traceability observability. What else can help this anonymous attendee thank you for your question do this quicker safer recovery? I I I think if I understand, you know, I I think what we have to know is what's the source of the problem. Uh I think with if you don't have certainty about that, then you tend to roll back a lot of things in the hopes that you capture you know the the the pro the underlying problem. My belief is that if you are working in smaller and smaller units and putting those units behind code units and putting those things behind flags, you have a greater chance to better understand if a specific change is causing that problem. In which case you can, it's a term that we use in the industry, you can kill that flag. You can turn off that negatively impacting experience while leaving everything else in place. If you don't work in those small units and you don't work behind flags and an incident is discovered and there are many releases between now and the incident, what can you do? You can try to triage and obviously try to get to the root cause. That means that pain will continue. It exists until you can figure that out. You can do a general roll back strategy which might cause its own friction because people were enjoying the use of something that suddenly might get taken away because we're not sure if that was causing a negative consequence. And so you kind of have to, you know, I would say you have to understand the changes. You have to control those individual changes more to be able to not get into that cycle of, you know, rolling everything back. >> Certainly as a reliability practice, if you are discovering issues relatively quickly after they happen, the roll back is the safest way to get into a known good state. Um you know we know that fixing forward can introduce all kinds of complications. Um but the argument is when you have feature flags um you can revert to a known state without changing any code just uh changing the feature flags. And I think in that state where you have had a number of deploys and an issue was latent and then it surfaced, right? Like you know, yes, a fixed forward may be your best option, right? But a lot then goes into resilience strategies, right? Like how are you making sure that it's only smaller blast radius of users who are impacted, right? Like how are you quantifying who those users are? Are you able to communicate with them? Right? There's so many things that go into a user's perception of how bad an outage was um that are not strictly related to the duration of the outage. >> It could be the impact, it could be financial, could be just how much it annoyed their day if they're the individual contributor that's using that tool. Uh all these matter and something we've been talking about the impact and speed of AI and just there's more of everything. But I think this is going to be the year where we're not just hearing we're going to hear about agents all the time, but I think we're going to hear about them also in the ops world because this agentic operations agentic is interesting because maybe a specific skill can be trained around this this threat posture that you have that if you reach a certain threshold or something, if that CPU spiked up like you said, Liz, maybe It's something that the AI will do something or at least help you with your root cause analysis because it can process more information. Buck stops with that developer that's in charge of the AI. But still maybe, >> right? Like I think this is one of the really cool and exciting things that we're starting to see our customers do with AI that we're doing ourselves. An AI doesn't care about being wrong. It can investigate six different parallel strands, right? Like whereas a human can probably only investigate one or two investigative hypotheses at the same time. Right? At the end of the day, someone does have to make a decision about what we are going to do about it and to feed that learning back into our software development process. But as far as being able to find a bug quickly that is starting to become faster for a machine to do than for a human to do, which is tremendously exciting. Mhm. >> I was going to build on that >> both the fact like kind of Liz said that if you've done multiple releases and not found the issue because I mean when we did big rollouts of of things you tend to be watching dashboards like a hawk and looking for any sign that something's not quite right. So you can spot that early and roll back quickly and then figure out the wise and wherefors um on that one. Um, but to that point, the other thought that popped into my head was I remember again, this is a GitHub memory, but when things went wrong, we did a lot of git bisect and things to try and work out what might have changed that causes things. And one of the really lovely things for me uh about AI tooling is that having joined a company relatively recently, I don't have to go through all of the git biceps and do git logs and try and work stuff out. I ask the AI to figure this stuff out for me in the codebase. Um, and that's way faster and to that point parallels. So we can we can help ourselves uh we can have it help us and and and do that amplification that you mentioned right at the start. >> Just to take one last question from the audience and it's kind of a big question but I think we can make it a little little and I think Ryan you kick off with this. Uh Rajie thank you for asking. How would you design a formal life cycle and architectural strategy to ensure that feature flags remain a temporary scaffolding rather than a permanent complexity? And how do you enforce the removal of these flags once a feature is 100% rolled out which that's very you said Ryan you alluded to that you were testing AI for this now because it can be like hey we think this is old which we love to stop zombie things from running environmentally and cloud cost but go more Orion. Yeah, it there's quite a few things in in that question, right? The first off, I I think you're getting to the difference between having a feature flagging system and having a feature management system. I think a feature flagging system is can be implemented in a variety of ways and you hear it you know whether uh folks are doing config files or whether they have database toggles or a variety of other mechanisms that create some code isolation and the ability to then change um you know the behavior of that code. Feature management in my definition means that you know flags do have a life cycle. There are flag types. There are long running flags, just to be clear, right? There are often shortlived flags to support feature releases, but there are long live patterns that make a lot of sense. Um, >> for sure, right? Like, you know, when we're talking about the difference, there's the difference between I have this code that I want to enable and disable. I'm doing a migration from path A to path B, right? Like, that's clearly a shortlived flag. >> You should be notified. You should have your agent rip out the old path when it's no longer needed. But also we have things like you know uh we call them control control rods like a nuclear reactor right like you know hey we need to emergency drop load right like you know that's not something that you want to take out of your codebase or here's the configuration for how many uh segments we process in parallel per worker right like again that's something that you might want to runtime tune >> knowing which one is which having that taxonomy having the hooks into your system to make sure that they are you know that they're able to control the roll out of the flag and also to receive the signal that the roll out is done and to rip out the code, right? Like that's kind of that direction that's going. >> Yeah. So once you have that ability to define flag types that your organizations are using, then you can define that life cycle around them, you know, how long should they should exist, where should they persist, what's the naming, what's something as simple as the naming convention of different types of flags to better understand their purpose uh inside my code. And so you want the ability to interrogate your system to understand what's operating in any given time inside your application and what's the state of that flag once it's discovered because an inventory of flags gives you nothing right it doesn't say if I have a 100 flags or a thousand flags I can't say whether I have a healthy implementation um it's what are those flags doing should they be there and if they don't have a purpose to the last part of your question How do we remove them? Now, I've heard or I've experienced a variety of mechanisms in terms of preventing that. I have customers that actually prevent developers from uh implementing new changes until they retire flags, right? They prevent actual pull requests from going through. It's pretty draconian. Um it's kind of hard-handed, but you know, that's that's a symptom. We're obviously trying to find nicer mechanisms, things to support the engineer in their effort of managing this because there's a lot of cognitive load associated with using flags. And so AI for us is uh you know for us uh you know what we think is is going to help an engineer the most in managing that life cycle and just saying hey kind of seems like this is no longer in use. You're the human in the loop. Can we remove it uh and take some action on your behalf? I think one other interesting piece that you discussed was the kind of validation and the knowing what the landscape of what's out there right if you are not attaching the state of each flag to each request right like then you are not getting valuable telemetry data about you know hey this outage might be affecting only users with this particular flag set right like so that operational data is really really important to feed into your processes >> we can talk about this all day. I think if we're going to end and wrap on a note, all of these we're talking about technical solutions to a sociotechnical problem. We all see great value in the developers, the operators, the humans in the tech processes and everything we're talking about is hopefully making your lives a little easier because that was supposed to be the idea of AI supposed to let us work on unique, interesting, novel problems. So hopefully we wish that to you and with the AI of things or not, progressive delivery is one of those best practices that will help you succeed, get fast feedback, and be better at your work. So we wish you luck and we'll see you in the lead dev slack if you need anything and we'll see you at the next one. Have a great day, y'all. Thank you so much, Liz, Claire, and Ryan. Thank you. >> Thank you. Bye everyone., Derisking Releases in the AI Era, YouTube, 01kmjraaj9fbph6amvep78j3xc.