Derisking Releases in the AI Era - CourtneyR.dev's 2nd Brain

Derisking Releases in the AI Era ![rw-book-cover](https://img.youtube.com/vi/3B4DG5PpBzA/maxresdefault.jpg) 🔗 **Source:** [Derisking Releases in the AI Era](https://www.youtube.com/watch?v=3B4DG5PpBzA) > [!note]+ Document Notes > tags: #progressive-delivery #feature-flags #AI #release-management #experimentation #observability #DevOps > author: Lead Dev > source: YouTube > reference: https://www.youtube.com/watch?v=3B4DG5PpBzA > date created: 2024-12-20 15:30 > > # 🏷️ Title: Derisking releases in the AI era >  ## 🧰 Action Inbox - [H1] Implement feature flag lifecycle management — prevent technical debt accumulation #feature-flags - [H2] Establish psychological safety for releases — reduce engineer fear of production deployments #team-culture - [H3] Set up quantitative measurement for faster feedback — enable data-driven release decisions #observability ## 🧭 Table of Contents - [[#📝 Summary]] - [[#📖 Main Content]] - [[#🗒️ Notes]] - [[#📚 References]] - [[#🗣️ Paraphrase]] - [[#📰 Newsletter]] - [[#✅ Action Items]] - [[#🧪 Hypotheses]] - [[#📊 Methodology]] - [[#❓ Questions]] - [[#🔮 Future Research]] - [[#💡 Implications]] - [[#🧠 Learning Styles Assessment]] - [[#📊 Accessibility Assessment]] - [[#🌍 DEIB Assessment]] - [[#✅ Logical Validity]] - [[#🛡️ DARVO Analysis]] - [[#🎯 Conclusion]] ### 🔗 References - [Harness Feature Flags](https://harness.io) - [Honeycomb Observability](https://honeycomb.io) - [Lead Dev Community](https://slack.theleaddev.com) ## 📝 Summary - AI adoption dramatically increases code complexity and release velocity, creating new deployment risks - Progressive delivery practices using feature flags, canary deployments, and experimentation can mitigate these risks - Organizations need psychological safety, quantitative measurement, and proper tooling to succeed with AI-amplified development - Human ownership and accountability remain critical even as AI generates more code - Proper feature flag lifecycle management prevents technical debt while enabling safer experimentation ## 📖 Main Content The panel discusses how AI acts as an amplifier for both productivity and risk in software development. AI adoption leads to 3-5x increase in lines of code initially, 30% more static analysis warnings, and 154% larger pull requests. However, these risks can be managed through progressive delivery practices that decouple deployment from release, enable targeted rollouts, and provide rapid feedback loops. The key insight is that while AI changes the scale and pace of development, the fundamental practices of small incremental changes, feature flags, observability, and experimentation remain the solution to managing deployment risk. ## 🗒️ Notes #### ⏱️ [00:00](https://youtu.be/3B4DG5PpBzA?t=0) — [H1] Panel introduction and AI impact overview - Key idea: AI dramatically increases code complexity and deployment velocity - Evidence: "AI adoption drives 3-5x increase in lines added in first month, 30% more static analysis warnings" #### ⏱️ [05:23](https://youtu.be/3B4DG5PpBzA?t=323) — [H2] Progressive delivery definition - Key idea: Rolling out smaller feature sets to subset of users with constraints on who and what - Evidence: "Progressive delivery is rolling out smaller feature sets to a subset of the audience" #### ⏱️ [08:45](https://youtu.be/3B4DG5PpBzA?t=525) — [H3] Psychological safety and release confidence - Key idea: Developer fear of production deployments slows velocity more than technical constraints - Evidence: "The developer feels the most risk at that point of release and de-risking that was really what unlocked velocity" #### ⏱️ [12:30](https://youtu.be/3B4DG5PpBzA?t=750) — [H2] Decoupling deployment from release - Key idea: Feature flags separate code deployment from feature activation - Evidence: "You can decouple a release of software from a deployment of software" #### ⏱️ [15:15](https://youtu.be/3B4DG5PpBzA?t=915) — [H3] Strategic user targeting for releases - Key idea: Target risk-tolerant users and customers with specific problems rather than random percentages - Evidence: "I like thinking first about who's more risk tolerant and then figuring out how you divide your users" #### ⏱️ [18:42](https://youtu.be/3B4DG5PpBzA?t=1122) — [H2] Progressive delivery as learning process - Key idea: Each increment provides different learning opportunities about functionality, scale, and performance - Evidence: "The release process can be a great learning process and provides opportunities for conversation" #### ⏱️ [22:18](https://youtu.be/3B4DG5PpBzA?t=1338) — [H3] AI as progressive delivery enabler - Key idea: AI can accelerate feature flag implementation and technical debt cleanup - Evidence: "Have the agent create the feature flag, have the agent wrap the code. It's not just a technical debt generation machine" #### ⏱️ [25:50](https://youtu.be/3B4DG5PpBzA?t=1550) — [H2] Blue-green deployments and infrastructure considerations - Key idea: Infrastructure as code makes blue-green more feasible by reducing permanent resource overhead - Evidence: "Infrastructure as code is making it cheaper because you can temporarily provision that environment" #### ⏱️ [28:33](https://youtu.be/3B4DG5PpBzA?t=1713) — [H1] AB testing and experimentation at AI scale - Key idea: AI development requires more experimentation due to multiple variables and parameters - Evidence: "Nobody seems to get those knobs and levers right the first time in AI development" #### ⏱️ [32:15](https://youtu.be/3B4DG5PpBzA?t=1935) — [H3] User interface churn risks - Key idea: Rapid AB testing can frustrate users if not carefully managed - Evidence: "Users do not like it when you move their cheese and being told you're in an AB experiment" #### ⏱️ [35:40](https://youtu.be/3B4DG5PpBzA?t=2140) — [H2] Quantitative measurement for faster feedback - Key idea: Metrics enable faster operational decisions than qualitative feedback alone - Evidence: "Quantitative measurement enables the fastest operational process" #### ⏱️ [38:25](https://youtu.be/3B4DG5PpBzA?t=2305) — [H1] Human ownership in AI development - Key idea: Developers remain accountable for AI-generated code and must own the delivery process - Evidence: "You build it, not necessarily write the code, but you build it, you kind of own it" #### ⏱️ [41:12](https://youtu.be/3B4DG5PpBzA?t=2472) — [H2] Safe experimentation requirements - Key idea: Responsible experimentation needs operational maturity including feature flags and observability - Evidence: "Responsible experimentation requires that you have a well functioning release process" #### ⏱️ [44:30](https://youtu.be/3B4DG5PpBzA?t=2670) — [H1] Feature flag lifecycle management - Key idea: Distinguish between short-lived release flags and long-lived operational controls - Evidence: "There are flag types. There are long running flags and shortlived flags to support feature releases" ## 📚 References - [Lead Dev Community Slack](https://slack.theleaddev.com) - [Honeycomb](https://honeycomb.io) - [Harness](https://harness.io) - [2025 DORA Report](https://dora.dev) ## 🗣️ Paraphrase - AI amplifies both development velocity and risk, requiring better progressive delivery practices to manage the increased complexity and larger blast radius of changes - Progressive delivery works by separating code deployment from feature activation, allowing teams to control who gets new features and when, reducing the risk of any single release - The human element remains critical in AI development - developers must maintain ownership and accountability for AI-generated code while leveraging quantitative measurement and proper tooling for safer experimentation ## 📰 Newsletter **Managing Release Risk in the Age of AI** As AI transforms how we write code, it's also dramatically increasing the complexity and velocity of our releases. This panel of experts from Honeycomb, Harness, and NA10 shared practical strategies for managing these new challenges. **Key Insight**: AI acts as an amplifier - it can generate 3-5x more code initially, but it also increases technical debt and deployment risk. The solution isn't to slow down, but to get better at progressive delivery practices. **What You Can Do Today**: - Implement feature flags to decouple deployment from release - Start with risk-tolerant users for new feature rollouts - Use quantitative metrics for faster feedback loops - Maintain human ownership even for AI-generated code The panelists emphasized that while AI changes the scale, the fundamental practices remain the same: small incremental changes, proper observability, and psychological safety for your engineering teams. Resources: [Learn WordPress](https://learn.wordpress.org/), [Developer Docs](https://developer.wordpress.org/), [WP-CLI](https://wp-cli.org/) ## ✅ Action Items #### DevRel - Developer advocacy: Create workshops on AI-safe development practices, demo progressive delivery with AI tools - Education: Develop curriculum on feature flag implementation and lifecycle management - Developer experience: Improve docs for AI-assisted development workflows - Community management: Foster discussions on AI deployment practices and safety #### OSPO - Policy & Governance: Establish guidelines for AI-generated code ownership and accountability - Security Management: Implement scanning for AI-generated code vulnerabilities - Developer Support: Provide training on progressive delivery practices for AI development - Strategic Alignment: Define KPIs for safe AI deployment practices #### Community - Maintainers: Support adoption of feature flag patterns in open source projects - Contributors: Create onboarding materials for AI-assisted development - Sponsors: Fund tooling for progressive delivery in open source ecosystems #### WordPress - Core: Implement feature flags for AI-powered features like block editor enhancements - Core Performance: Use progressive rollouts for performance optimizations generated by AI - Core AI: Establish experimentation framework for AI feature development - Playground: Create sandbox environments for testing AI-generated code safely - Training: Develop lesson plans on progressive delivery for WordPress developers ## 🧪 Hypotheses - Organizations with mature progressive delivery practices will see less disruption from AI adoption - Feature flag lifecycle management prevents accumulation of technical debt from rapid AI development - Quantitative measurement systems enable faster identification of AI-generated code issues - Teams with psychological safety around releases will adopt AI tools more successfully ### 📊 Methodology - Study design: Compare deployment metrics before and after AI adoption across organizations with different progressive delivery maturity - Data sources: Release frequency, rollback rates, incident response times, developer satisfaction surveys - Experiments: A/B test different feature flag strategies for AI-generated features ### ❓ Questions - How can we better balance AI development velocity with code quality and maintainability? - What metrics best predict the success of AI-assisted progressive delivery implementations? - How do we prevent AI from creating unsustainable technical debt in feature flag systems? ### 🔮 Future Research - Investigation of agentic operations for automated incident response and root cause analysis - Development of AI-powered feature flag lifecycle management systems - Study of psychological safety impacts on AI adoption in engineering teams ### 💡 Implications - Practice: Teams need to invest in progressive delivery tooling before scaling AI adoption - Policy: Organizations must establish clear ownership and accountability for AI-generated code - Security: AI amplifies the blast radius of security vulnerabilities, requiring better isolation - Education: Engineering curricula must include progressive delivery and experimentation practices - Funding: Budget allocation should prioritize observability and feature management infrastructure ### 🧠 Learning Styles Assessment Reading level estimate: 12th grade #### 👀 Visual learners - Present: Panel discussion format with speaker identification - Improve: Architecture diagrams showing feature flag deployment patterns, visual comparison of traditional vs progressive delivery #### 🔊 Aural learners - Present: Full audio discussion with expert insights - Improve: Podcast-style summary of key concepts, audio transcription quality #### 🗣️ Verbal learners - Present: Clear definitions of progressive delivery, feature flags, and related concepts - Improve: Glossary of terms, analogies comparing deployment strategies #### 🧩 Kinesthetic learners - Present: Practical examples and use cases from real implementations - Improve: Hands-on tutorials for feature flag implementation, step-by-step deployment guides #### 🧠 Logical learners - Present: Statistical evidence of AI impact, structured discussion of practices - Improve: Decision trees for release strategies, quantified risk assessment frameworks #### 🧑‍🤝‍🧑 Social learners - Present: Panel discussion format with multiple perspectives - Improve: Community forum integration, peer learning opportunities #### 🧘‍♀️ Solitary learners - Present: Individual reflection prompts implicit in the content - Improve: Self-assessment checklist for progressive delivery readiness #### 🌿 Naturalistic learners - Present: Discussion of ecosystem impacts and sustainable practices - Improve: Environmental considerations of increased computing from AI development ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Speaker names clearly identified, structured conversation flow - Improve: Enhanced heading structure for navigation, speaker role descriptions #### Keyboard navigation - Present: Standard video player controls available - Improve: Timestamped navigation shortcuts, transcript-to-video synchronization #### Contrast and readability - Present: Video format with clear audio - Improve: High-contrast transcript formatting, adjustable text size options #### Media and captions - Present: Full transcript provided - Improve: Synchronized captions with speaker identification, downloadable transcript ### 🌍 DEIB Assessment #### Cultural breadth - Present: Global technology perspective from panelists with international experience - Improve: Examples from diverse organizational contexts, non-Western deployment practices #### Inclusive language - Present: Professional technical terminology with explanations - Improve: Plain language alternatives for complex concepts, cultural context awareness #### Neurodiversity support - Present: Structured discussion with clear topic transitions - Improve: Visual topic indicators, content summary at regular intervals #### Representation - Present: Diverse panelist backgrounds and company perspectives - Improve: Balanced gender representation, varied organizational size examples ### ✅ Logical Validity #### Strong reasoning - Deductive strength: Clear progression from AI impact premises to progressive delivery conclusions - Inductive strength: Multiple real-world examples supporting general principles - Causal reasoning: Well-established links between deployment practices and risk reduction - Analogical strength: Effective comparison between traditional and AI-amplified development #### Fallacies - Appeal to authority: Some reliance on expert opinion without supporting data - Hasty generalization: Limited sample size of organizational examples #### Tone of voice - Overall tone: Professional and collaborative with practical focus on problem-solving ### 🛡️ DARVO Analysis - Deny: No patterns of denial identified - Attack: No attacking behavior observed - Reverse victim/offender: Not applicable to technical discussion - Emotional cues and rhetoric: Positive, solution-focused language throughout ## 🎯 Conclusion The panel demonstrates that while AI dramatically amplifies both development velocity and deployment risk, the solution lies in doubling down on proven progressive delivery practices rather than slowing development. The key insight is that AI doesn't fundamentally change the need for feature flags, observability, and experimentation - it makes these practices more critical. Organizations that invest in progressive delivery infrastructure and psychological safety will be best positioned to leverage AI's benefits while managing its risks. The human element remains crucial for ownership, learning, and strategic decision-making even as AI handles more of the code generation. ## 🧱 Refactor Appendix ### 📚 Literature Split Pad ### [H1] Progressive delivery reduces AI deployment risk > [!quote] [Derisking releases in the AI era](https://www.youtube.com/watch?v=3B4DG5PpBzA) by Lead Dev via YouTube. Published 2024. Accessed 2024-12-20. > "Progressive delivery is rolling out smaller feature sets to a subset of the audience with constraints on who and what." Foundation practice for managing increased complexity from AI-generated code. [view highlight](https://youtu.be/3B4DG5PpBzA?t=323# --- ## Full Transcript [00:12] Hello everyone and I just huge thank you [00:16] to everyone for joining us today for [00:18] today's panel. We're talking about [00:20] d-risking releases at the speed and [00:23] impact of AI. This is a lead dev online [00:27] panel discussion created in partnership [00:29] with harness. Discussion is going to [00:31] last about 45 minutes after which both [00:34] myself and the panelists will head over [00:37] to the lead dev slack. uh if you have [00:40] any questions you can ask throughout [00:44] here and then if they continue or if you [00:48] want to continue the conversation that's [00:50] a wonderful place to do is the hashtag [00:53] panel Q&A. Uh for those joining us live, [00:57] I'm going to share in the chat right now [00:59] the link to that lead dev and I will add [01:02] as a resource because it's not ephemeral [01:05] and or it is ephemeral on Zoom, but it's [01:08] a really great community and I do [01:10] recommend joining it. I like a Slack [01:13] community. I don't know how others. So [01:15] yeah. Uh so we'll answer questions if [01:17] they see fit throughout. Otherwise, like [01:20] I said, we'll continue on there. So you [01:22] have the Q&A tool. Please use it. So, [01:24] let's get started with some [01:25] introductions. I'm Jennifer Rigggins. [01:27] I'm a technology journalist. You may [01:30] have seen me host a lead dev panel [01:32] before. And today, I'll be joined by [01:35] three lovely panelists that I invite [01:37] each of you to introduce yourselves. [01:40] Claire, you are at the top of my screen, [01:41] so I let you go first. [01:44] >> Thank you very much, Jennifer. Um, [01:46] longtime lead dev fan right from the [01:48] early days. So, lovely to be on this [01:50] panel. Um, currently director of [01:52] engineering at NAN, heading up all of [01:54] the enterprise side of uh, engineering [01:56] there. Um, spent a lot of years working [02:00] in tech um, across the globe um, but [02:04] based in the UK um, and some of you may [02:06] have seen me more recently on panels uh, [02:09] representing Netlefi. Um, recently [02:11] joined NA10 for a new scaleup challenge. [02:14] Um, I'll maybe ping that over to Liz [02:18] now. [02:20] Hi, I'm Liz. I'm a technical fellow at [02:22] Honeycomb and I previously worked for [02:25] Google as an Nesserie for something like [02:27] 11 years. Um, so I've been around this [02:30] block of uh tension between release [02:33] velocity and reliability for years and [02:35] years and years and I'm very excited for [02:37] what the future holds. Ryan [02:40] >> and hi everyone. My name is Ryan Ryan [02:43] VA. I'm a feature management and [02:45] experimentation advisor advisory [02:47] director here at harness. Uh I have the [02:50] opportunity to work across a broad broad [02:52] spectrum of uh organizations who are [02:56] implementing uh feature level [02:58] observability within their groups and [03:01] I'm excited to to share you know my [03:03] experiences working with them with you. [03:06] Okay, just a little background as people [03:08] are jumping on. They came to hear from [03:10] y'all. But at least in open source, the [03:12] use of generative AI directly drives a [03:15] three to five multitude increase in [03:18] lines added in the first adoption month [03:21] in projects. This gains dissipate after [03:24] two months. It's not a bad thing. uh [03:27] lines of code not usually a great metric [03:31] for anything but I think it's actually a [03:32] pretty reliable predictor of complexity. [03:35] So while we don't want to measure lines [03:38] of code in individual level for various [03:40] job safety and security and [03:42] psychological safety reasons [03:44] it's not bad one to measure of your [03:46] complexity of your database. So also [03:50] it's been observed that persistent [03:51] technical debt accumulation static [03:54] analysis warnings have increased by 30% [03:57] and code complexity increases by 41% [04:00] post adoption of AI. The 2025 Door [04:04] report, which I'm a journalist, I love [04:07] these percentages and things, said that [04:10] AI tends to increase pull request size [04:13] by 154%, [04:15] which of course is [04:18] increasing dramatically that blast [04:20] radius of any release. And I think Doris [04:24] said it very well that it called AI an [04:27] amplifier and a multiplier. So [04:32] honestly, I don't think AI changes [04:36] the things because if you were doing the [04:38] DevOps and engineering best practices [04:40] and guardrails that we've been talking [04:41] about for years, AI, you should be [04:45] logically able to keep up with this new [04:48] pace. You probably also aren't releasing [04:50] as much. You are not adding that many [04:53] lines of code because your quality [04:55] checks are in place and things. But it [04:58] is if you don't have those in place and [05:01] you don't have these progressive [05:03] delivery best practices that we're going [05:04] to talk about in place. Well, [05:08] every release runs a certain risk, but [05:10] the larger release and especially those [05:13] written without the checks of human [05:14] beings in the loop run the risk of more [05:17] damage to your reliability, security, [05:19] reputation, bottom line. So, we're on [05:23] the call. Spoiler alert. [05:27] We're all big fans of this idea of [05:28] breaking work into smaller manageable [05:30] chunks reduces risk. It's not an [05:33] original idea, but this idea of then [05:36] progressive delivery practice that can [05:38] reduce that blast radius more. And these [05:42] are just so [05:45] much more important in the face of [05:48] managing deployments with unpredictable [05:51] AI elements are involved. So yeah, [05:54] that's our topic. That's what we're [05:57] talking about today. So we can help you [05:59] figure out how to have the smaller, more [06:01] manageable [06:03] releases and the smaller man more [06:06] manageable code and impact. And I know [06:08] from the questions ahead of time, there [06:10] was specifically people looking for how [06:12] to do that under this idea of enterprise [06:16] breath and complexity of code. So [06:19] without further ado, I like a [06:21] definition. to start off. Claire, would [06:24] you like to define what you think [06:26] progressive delivery is? [06:30] >> I think there's people that have defined [06:31] it better than I can, but I can [06:33] certainly describe what I would expect [06:35] as as to happen. Um, so for me, [06:37] progressive delivery is um [06:41] rolling out smaller feature sets to a [06:44] subset of the audience. Uh so you can [06:46] constrain it in two different ways. uh [06:48] as in who who is affected and how much [06:50] of the new thing you're pushing out [06:52] there and whether you're running things [06:53] in parallel commonly done with with [06:56] feature flags and uh canary rollouts and [06:58] things like that but I don't necessarily [07:00] want to uh dig into to specifics like [07:03] that right now I think the bigger the [07:05] bigger conversations more about how we [07:07] wrangle all of this [07:09] >> and then something you did mention about [07:11] progressive delivery that I thought was [07:12] very interesting that there's a [07:14] correlation between companies companies [07:17] that do have the practice of progressive [07:20] delivery and like positive work [07:22] environment. [07:23] >> Yeah. So I've I was uh a big fan of [07:28] feature flags from when I first [07:29] encountered them at GitHub um a long [07:31] time ago now, many many years ago. Um, [07:36] but I have definitely then seen other [07:39] companies that they're just not even [07:40] aware of them or why you want to use [07:42] them or having the maturity to to design [07:45] um [07:47] u changes that can take advantage of [07:50] them. Um, I think that this is to the [07:53] point you're making that this ties to me [07:55] the trrimodal nature of tech companies [07:58] um where you've always got people who [08:01] are pushing the bleeding edge. um [08:03] whether that's AI or anything else, but [08:05] it's adopting the good practices. It's [08:06] doing the DevOps is doing all of the [08:08] things we've been talking about for a [08:09] while now. Then you've got the kind of [08:11] leaders within countries that are not [08:13] necessarily [08:15] completely there but are doing their [08:16] best and trying. And then you've, you [08:18] know, some people on cloud migration [08:20] still okay, you know, how long have we [08:21] had cloud? They're the kind of third [08:23] tier of ones and they're the ones that [08:26] even now I would suspect don't have the [08:28] the organizational capability or even [08:30] the infrastructure capability to do [08:32] progressive rollouts. This is like back [08:34] when I started with gold CDs being [08:36] pressed for your master version of the [08:38] software that you released. Just dating [08:40] myself slightly [08:45] >> Ryan. Yeah. What do you want to add to [08:47] that on the psychological safety? Oh [08:50] yeah. Yeah. You know, um, you know, [08:52] progressive delivery is a is a is a very [08:54] interesting topic to me. I've been [08:56] preaching it for a very long time. Um, [08:59] specifically around using obviously, you [09:01] know, feature flags to kind of support [09:03] that type behavior, although there's [09:04] other mechanisms to to do that as well. [09:07] But, um, I was just thinking about [09:09] psychological safety. I was thinking of [09:10] a conversation that I had um, with a VP [09:14] of engineering at one of the largest [09:16] credit bureaus, you know, in the world. [09:18] and they were talking about kind of [09:20] their efforts around application [09:22] modernization which is also kind of a [09:23] buzz word and you hear a lot of people [09:24] talking about it. Um [09:27] they were very interested in kind of [09:29] helping their engineering workforce [09:31] modernize the tools that they were [09:32] using. You know they were using kind of [09:34] legacy application legacy languages. [09:36] They wanted to to take advantage of you [09:39] know more modern architectures. uh and [09:41] they thought as an organization that if [09:43] they did this did this modernization [09:46] they would see an acceleration then in [09:48] um new capabilities coming to the market [09:51] right just a higher velocity rate. What [09:53] they discovered was that they did [09:56] modernize their tech stack and they were [09:57] using more modern architectures. uh [09:59] unfortunately it was having the [10:01] unforeseen consequence that a lot of the [10:03] testing was shifting you know is they [10:05] were shifting to more testing uh within [10:08] development because they didn't have as [10:09] much experience working with these [10:11] modern technologies and stacks and so [10:13] they felt like they had to compensate [10:14] with testing more and so actually things [10:17] slow down as they as they kind of ramped [10:20] and adopted these new technologies. It [10:22] wasn't until they started using feature [10:23] flags, they kind of created that safety [10:25] net that they then saw an acceleration [10:28] because psychologically for the [10:30] engineers there was less risk in that [10:32] release process. So great that they're [10:35] you know now serverless or you know [10:37] using lambdas or micros service or [10:40] microrends and all those sorts of things [10:42] but ultimately the developer feels the [10:45] most risk at that point of release and [10:47] de-risking that was really kind of what [10:50] unlocked it for them in terms of you [10:52] know feeling safe to to get ideas out [10:55] faster. All systems are sociotechnical [10:58] systems, not just the technical system, [11:00] right? Like no technological change is [11:03] actually going to make a difference if [11:04] your team is afraid to interact with [11:07] production in any way because they, you [11:08] know, it pages them all the time or they [11:11] get yelled at every time they cause an [11:12] outage. [11:15] >> So we have feature flags. [11:18] Maybe we want to quick to find that in [11:20] like a few words and then let's try to [11:22] run popcorn. [11:23] >> I think we should back up a step though, [11:25] right? like you know when CLA was [11:27] talking earlier about this idea of um [11:30] rolling out features to a percentage of [11:31] users right like I think even going [11:34] beyond a step beyond that the critical [11:37] insight that progressive delivery had [11:39] was that you can decouple a release of [11:42] software uh from a deployment of [11:44] software right um so when we are [11:49] enabling the ability to turn on specific [11:52] functions of our code at times of our [11:53] choosing rather than at times are not [11:55] choosing. I think that is the key thing [11:58] that then enables teams to better manage [12:00] their own releases, right? Like and then [12:02] we can talk about the details of feature [12:03] flags from there. [12:06] >> And then when we say roll out, for those [12:09] that aren't as familiar with this [12:11] concept which James Governor kind of [12:12] coined as progressive delivery, how do [12:16] you decide [12:18] who to release to or not? [12:23] Certainly people um do everything from, [12:26] you know, hey, we're going to roll out [12:27] just to 1% of users at random. There are [12:30] also circumstances where you might have [12:33] early access customers, right? Like you [12:35] might have customers that have opted in. [12:37] Um we actually just completed a pretty [12:41] long process to um turn on a new metrics [12:44] back end at Honeycomb. And we had [12:46] customers that wanted to opt in early. [12:49] And then we had customers that were [12:50] like, you know, I don't want to be [12:51] broken, right? Like, you know, please [12:52] don't please don't give this to me [12:54] early, right? Like, and that self- [12:56] selection was really, really important [12:57] for uh people to get that um early [13:02] access if they were more risk tolerant [13:04] um and they wanted to influence our [13:05] future development and for pe for us to [13:08] not break people that wanted to be a [13:10] little bit more conservative. Um, and [13:12] then, you know, there are also the [13:14] people who decide, you know, these users [13:16] are free tier. I'm just going to [13:17] experiment on them, right? Like they're [13:18] not paying me any money, which is a fair [13:21] business choice potentially. Um, but I I [13:23] like thinking first about, you know, [13:25] who's more risk tolerant and then kind [13:27] of figuring out, you know, how you [13:29] divide your users into risk tolerant and [13:30] not risk tolerant from there. There can [13:32] also be building on that um [13:36] uh customers that maybe on paper are not [13:38] risk tolerant in general, but they've [13:40] had an issue with something and you're [13:42] trying to roll out something that helps [13:43] solve their problem. Um and most of the [13:46] time in most of the businesses where [13:48] I've had direct communication with the [13:50] customer and these things, they're like, [13:51] "Yeah, sure. We'll try it out. It might [13:53] solve our problem. We appreciate you [13:54] taking us seriously." And it is actually [13:56] helpful to build relationships um that [13:59] way, which is to me an extra bonus on [14:03] then making sure that the person who's [14:05] got the real world problem gets the fix [14:06] properly tested as well. [14:09] >> Yeah. And that always jumps out at me is [14:12] that that that's really well said, [14:14] right? And that relationship building [14:16] because to me the release process can be [14:18] a great learning process. And if you do [14:21] it progressively, if you do it [14:23] incrementally, if you target how you [14:26] release things, it provides [14:28] opportunities for conversation, right? [14:31] It provides an easy opportunity to say, [14:33] is this working? Does it do the thing [14:35] that you wanted it to do? Is it conflict [14:38] with anything that you previously were [14:40] doing? Um, and you really get to enhance [14:44] your ability to align, you know, on the [14:47] outcome that you're delivering with your [14:49] customer, right? You can really, you can [14:51] do some course correction, right? If [14:53] you're not just turning it on for [14:54] everyone and, you know, getting some [14:56] feedback and figuring out, oh well, [14:58] maybe there's an iteration on this that [15:00] we need to do before we kind of go a [15:02] little bit more broad around it. And so [15:05] to me, progressive is, [15:08] you know, there's the incrementing [15:10] component of progressive, but it's [15:12] really what can I learn at those [15:14] increments and can I learn something [15:16] different? Like one might be functional, [15:19] another moment in that progressive [15:20] delivery might be more around scale and [15:23] reliability, right? As I increase [15:26] >> and performance, right? Like you know, [15:27] if you see 10% CPU increase from turning [15:29] it on for 1% of your users, you go, hm, [15:32] okay, right? like that needs a little [15:33] bit more work. [15:35] >> Exactly. So, you know, you can learn [15:37] different things along the way and [15:40] that's what opens up your mind in terms [15:42] of deciding, you know, do I go 1% or 10% [15:45] or do I go, you know, a targeted list of [15:47] alpha beta customers and then into a [15:50] percentage or maybe I never do a [15:51] percentage, but I'm, you know, I I work [15:54] with my community to get feedback at [15:56] every step along the way. [15:57] >> Right. Exactly. Self- opting into [15:59] feature flags is also really really [16:00] awesome, right? like you know every time [16:02] you have an experimental or beta slider [16:04] inside of your product right like that's [16:05] something that you should also be [16:06] managing through your feature flag [16:07] framework but I think to Jen's original [16:10] point about uh progressive delivery and [16:12] about splitting up changes into smaller [16:13] increments right like when we're talking [16:15] about splitting up into smaller [16:16] increments we are not just referring to [16:18] the code right like I think that is the [16:19] critical thing right like we are trying [16:21] to get feedback faster accelerate our [16:24] learning cycle not just by shipping code [16:26] faster and and in smaller increments but [16:29] also um you know microlearning from [16:31] smaller numbers of users faster rather [16:33] than you know waiting to deploy it to [16:35] everyone before we realize oh we built [16:37] the wrong thing. [16:39] This is very interesting because usually [16:41] the example for progressive delivery is [16:44] roll out to well Australia or or [16:51] >> hey stop picking on Australia. We may be [16:54] very few people and we may be in a [16:56] strange time zone but like you know [16:58] we're real users. [17:00] >> Yeah. And also the idea is like you're [17:02] not being that responsive if it's not [17:05] when your most of your staff is working. [17:09] If you're zero or another Aussie company [17:11] and you want to release to Australia [17:13] first that's that's a choice maybe [17:15] logical but I think these ideas of [17:17] thinking about it more strategically as [17:19] you said Ryan to have a conversation is [17:21] better but on the other hand you can [17:24] dump it on people that are not worthy. [17:26] uh back when James Governor I feel like [17:30] it was 2017 but I don't know first was [17:32] introducing this concept or the concept [17:35] existed but like coalesing around this [17:38] term progressive delivery he said like [17:40] one client he knew did dumped everything [17:44] first on the people that used illegal or [17:47] stolen credit or fraudulent credit cards [17:50] as the first to me that that's lower [17:53] risk and I get that and if you're going [17:56] to risk they're not the customers you're [17:57] really worried about. I guess on the [17:59] other hand, it doesn't really show to me [18:01] that you care about what you release [18:03] that much and you want that feedback. [18:06] It's really just is this going to work [18:08] or not? Not this is a beautiful feature [18:10] we are bequeething to others. [18:13] >> Honestly, I think more teams where [18:16] possible should dog food, right? like [18:18] you know right like Honeycomb's [18:20] deployment cycle we automatically deploy [18:23] whatever is at at main once an hour [18:27] >> but it goes through phases of first [18:29] every Honeycomb employee gets it for our [18:31] internal uh instance of honeycomb before [18:33] it goes out to production right so that [18:36] also is a a form of progressive delivery [18:38] right like you know yes it's progressing [18:41] through environments but like they're [18:42] not throwaway environments they're real [18:44] environments that our employees use and [18:46] we will scream if we notice something [18:47] wrong. [18:49] >> That's the platform engineering way. [18:51] You're the worst and the best customers. [18:53] Best tightest feedback, Luke, Claire. [18:55] >> Yeah, we we really are the best and the [18:56] worst customers as developers. I mean, [18:58] we we did that at at GitHub. There was a [19:01] staff bar and things were enabled there. [19:03] Plus, to Liz's earlier point, we we had [19:05] it was possible for customers to opt [19:07] into certain feature flags themselves [19:08] when they chose to as well. Um, again, [19:11] at Notify now at NA10, we're huge users [19:14] of our own product. uh even if what it [19:17] takes to build the product isn't exactly [19:18] what you'd necessarily use uh for the [19:22] actual say software development life [19:24] cycle. Um but we we use it for all sorts [19:26] of things internally because then and we [19:28] have what we call the internal instance [19:30] which um to Liz's point yeah that does [19:34] spot a lot of things before they hit the [19:36] wider audiences. [19:38] I think one interesting AI tiein here is [19:41] that we've talked a lot about how AI [19:43] puts stress on the system, [19:45] >> but I also see AI as an opportunity to [19:48] make it easier to do progressive [19:49] delivery in a lot of ways, you know, [19:51] >> right? Like development teams struggle [19:54] when you tell them, you know, hey, can [19:55] you write like, you know, can you put a [19:56] feature flag around this? And they're [19:57] like, I don't know how to do this, [19:59] right? Like [20:01] >> write a skill, have the agent create the [20:03] feature flag, have the agent wrap the [20:05] code. Done, right? Like it's not just a [20:07] technical debt generation machine. It [20:09] can also be a technical debt paying down [20:10] machine. [20:11] >> Absolutely. [20:12] >> Yeah. And and you know AI is also going [20:15] to support that debt elimination, right? [20:17] The idea of understanding what's in use [20:20] inside your application as it relates to [20:22] a future flag. Is it being called? Is it [20:24] routing traffic? Is it no longer being [20:26] modified? You know, and AI, you know, [20:29] we're investigating ways that AI can [20:31] interrogate your flags and say, are you [20:34] worthy of still existing? Right? And if [20:36] not, we're going to we're going to do a [20:38] pull request uh and we're going to [20:40] suggest that you get removed. Right? So [20:43] all all stages of the life cycle can be [20:45] affected by AI as it relates to flag [20:48] management in our opinion. [20:51] >> Also, what about other things? We're [20:53] talking about feature flags, but there's [20:54] so much more to [20:57] progressive delivery. What about has [20:59] anyone used blue green or AB testing [21:03] where you aren't really sure where you [21:06] want to go and which is the best feature [21:08] to solve this problem. How are you going [21:10] about that? [21:16] I think that with regards to blue green [21:18] for infrastructure like I think that [21:20] that is [21:22] a good thing to do if you can afford to [21:25] do it. Um I think a lot of the challenge [21:27] of blue green is that it does require [21:29] doubling your resource footprint which [21:30] may not be palatable at internet scale. [21:33] Um [21:35] however infrastructure as code is making [21:37] it cheaper to do that because you can [21:39] certainly you know temporarily provision [21:42] that uh that blue green environment uh [21:45] during the uh overlap period and then [21:46] decommission the old one whenever you're [21:48] satisfied. Whereas previously it was a [21:50] okay now we have to keep a blue and a [21:52] green running permanently. That's not so [21:53] great. Um, [21:55] >> oh yeah. [21:56] >> I I think with regards to the [21:59] other thing that you suggested, I I [22:01] think, you know, we definitely can [22:07] I I I think I think it's definitely [22:09] worth worth doing. I I Yeah. [22:12] >> Yeah. Okay. [22:13] >> It it it it is um I think [22:16] experimentation was an odd word to bring [22:19] up with product engineering groups 10 [22:21] years ago. Um, I think more and more I [22:26] see engineering leaders adopting it as a [22:30] key way to, you know, quantitatively [22:32] measure the impact of of what they're [22:34] doing. Uh, it's not a replacement for [22:36] qualitative engagement. Um, but it can [22:39] enhance your overall again that learning [22:41] experience. And I think AB testing I see [22:43] it more and more being adopted as a [22:46] pattern within engineering to you know [22:50] answer answer questions and and answer [22:52] them quickly. You know if we tie this [22:54] back to AI um you know in AI development [22:59] I always say that there's a lot of knobs [23:01] and levers in AI. You know there's um [23:05] things like uh models and there's [23:07] temperatures and parameters that you [23:09] want to adjust. there's prompts that you [23:11] feed into it. Um, nobody seems to get [23:15] those knobs and levers right the first [23:17] time. And so the ability to quickly um [23:22] change any of those individual [23:24] components and then have a feedback loop [23:27] in terms of what's the outcomes then [23:30] from that adjustment. Am I seeing less [23:32] hallucination? am I seeing better, you [23:35] know, outputs from, let's say, if I'm [23:37] doing a Gen AI, better outputs. [23:40] That's very informative. And so I think [23:42] in the in the AI world, [23:46] it's underlining the need to do AB [23:49] testing, experimentation quickly. Um, [23:52] >> and in fact, the AI world has a term for [23:53] this, right? Like it's evs, right? Like [23:55] you want to constantly doing EVELs, [23:56] right? Like you know, no one disagrees [23:58] with that. I here's the risk that I for [24:00] that I do foresee with regard to AB [24:01] testing, right? [24:03] Yes, it's good when it's invisible to [24:05] the user. [24:06] >> I worry that with the pace of AI [24:08] development, the bottleneck becomes the [24:11] rate at which you can churn user [24:14] interface and right like users do not [24:16] like it when you move their cheese and [24:18] being told you're in an AB experiment [24:20] test group, right? Like you know, and we [24:22] moved their cheese to see what how you [24:23] reacted to it, right? Like at some point [24:25] it grates on the user. Um, so I think [24:28] we're people are going to have to figure [24:30] out how to prioritize, right? Like doing [24:32] these AB tests and not testing every [24:34] feature on, you know, the same set of [24:36] unsuspecting users because they will get [24:38] very unhappy when you keep turnurning [24:39] their UI. [24:40] >> Yeah, there's an interesting tension [24:42] there for me with the AI side because [24:44] you've got the the constantly changing [24:46] churn and yes, I've had enterprise [24:48] customers at previous jobs say, "Why is [24:49] the interface changing?" And you're [24:50] like, "It's only changed once in a [24:51] year." And they're like, "No, not [24:53] happy." Um, but we're also like there's [24:56] the hyper localized hyperpersonal [24:58] software that's being built at speed and [24:59] like where does this all be play out [25:02] over time. I think I don't have answers [25:04] but I think it's fascinating to live in [25:07] it and see what's going on to be honest. [25:09] Um [25:11] where I would also say there something [25:13] we haven't mentioned here where AI can [25:14] help with the experimentation is I've [25:16] got some of that happening where we are [25:17] now at NA and a couple of my engineers [25:19] have actually used the AI around the [25:21] analysis and around what to try next and [25:23] what to tweak and and kind of use it as [25:25] a investigative tool um rather than a [25:28] hit let's just build a new UI or build [25:30] or whatever and I we've found out some [25:32] interesting facts that way that we might [25:33] not have stumbled across otherwise which [25:35] is an interesting angle. Can you give an [25:37] example of like what you found that way [25:39] or how that worked? [25:41] >> Um, [25:43] we were trying to because obviously when [25:46] you run experiments, you kind of if [25:47] you're doing it right, you're trying to [25:48] figure out what is the is this good or [25:50] bad? did we move a needle? [25:52] The needle wasn't moving in the way we [25:54] expected. And instead of just being able [25:57] to shrug our shoulders, it enabled the [25:59] team to do much more of an in-depth [26:00] investigation into that and then come up [26:03] with some possible causes based on that [26:05] which they are now running further [26:07] experiments on. So I don't have uh [26:09] complete conclusions there. But it it [26:11] opened up roots that we might not have [26:13] thought about taking or like prompted us [26:15] to think about it more deeply. and the [26:17] engineers took this on themselves and [26:18] were like, "No, I need to understand [26:19] this better and and just really hammered [26:21] it and I I loved that." [26:25] >> One of the areas around progressive [26:28] delivery, we're talking about like we've [26:30] been talking about lowering and limiting [26:32] blast radius. [26:34] The other side of that is because of the [26:36] rapid feedback and that's something that [26:39] both AI can make better. It's known for [26:42] that because there's the prototyping, [26:44] there's different options, but also if [26:47] your PR site's getting longer or the [26:50] bottleneck was never writing code. So if [26:53] as it moves down the line, there's a [26:56] slower feedback that way or maybe even [26:59] this idea that developers didn't create [27:02] the code, so they're waiting for [27:04] feedback and they don't fully [27:05] understand. So, what are ways that we [27:07] can tighten this feedback cycle with [27:09] faster releases when we're releasing so [27:13] much more? [27:18] >> I I think that um I think you have to [27:22] lean on quantitative measurement a bit [27:27] um to get a faster feedback loop. Um I [27:31] think picking up the phone is still the [27:33] fastest, right? engaging with your [27:34] customers closely on individual changes. [27:36] But unfortunately, that's also kind of a [27:38] little more anecdotal in nature and [27:41] you'd be hesitant to take strong action [27:44] based on a couple of data points. [27:46] Whereas, if you can measure things [27:48] quantitatively, you should have greater [27:51] confidence in behavior or action as a [27:54] subsequent, you know, you know, [27:55] subsequently from the signal that's [27:57] delivered, right? I saw things improve [27:58] or I saw things degrade or I didn't [28:00] really see any impact on significant [28:02] metrics. And so um I think quantitative [28:06] measurement enables the fastest [28:09] operational process. Um immediate [28:12] feedback though you you can still attain [28:15] you know qualitatively. [28:17] I think there is one interesting [28:19] challenge which is we know that language [28:21] models do best when they get immediate [28:23] feedback right like you know when they [28:24] get immediate feedback on your tests are [28:26] not passing right and how do we connect [28:28] that loop back with production with what [28:30] users are seeing for reliability signals [28:33] right like you know we should be [28:34] connecting them to our observability [28:36] tooling albeit you know that might be on [28:37] a delay as far as you know how often you [28:39] release right like you know it might [28:41] take half an hour or an hour you know or [28:43] heaven forbid you know 8 hours for that [28:45] for that to go out right like and then [28:46] for you to be able to feed that signal [28:48] back into language model of this is how [28:50] it's actually performing and then I [28:51] think you know things like amplitude mix [28:53] panel right like things that are for [28:55] user analytics are another place where [28:58] you might want to feed that into [28:59] language model although that starts [29:01] getting in the domain of user [29:02] requirements which I think is more of a [29:03] human consideration rather than a uh [29:06] thing that a language model can directly [29:07] do something about [29:09] >> yeah I think the the struggle that [29:10] you'll have on the observability side is [29:14] that you know it tends to not be [29:17] cognizant of maybe individual changes [29:19] that might be occurring inside the [29:21] application or you know you've got a [29:23] situation where of course many things [29:25] are constantly changing inside the [29:27] application whether it's you know other [29:29] other new features being introduced or [29:32] new capabilities or new infrastructure [29:33] or new networking or new security what [29:36] have you. there's a whole multitude of [29:38] things and so observability tries to [29:41] take all that in and tries to find the [29:43] problem you know that that's in the [29:45] noise uh and you know I think having a [29:49] better sense of feature level [29:50] observability so changing the paradigm [29:53] from measuring the application to [29:55] measuring the individual change uh is [29:57] important to you know isolate that [30:00] signal faster to take that action [30:07] Okay, something we did talk about was [30:12] this sense of ownership in the age of [30:14] AI. [30:15] So [30:17] even if AI is involved in or wholly [30:20] creating things, [30:22] who is in charge of the delivery of that [30:26] thing and who owns it? [30:30] >> Definitely been a big one for me. Um [30:34] I'm [30:35] very strongly of the belief is you build [30:37] it, not necessarily write the code, but [30:39] you build it, you kind of own it type of [30:42] um behaviors. Uh I know there's various [30:46] or or structures that tweak that, but by [30:49] that I mean I don't mind which tools you [30:51] use. If you want to tap it out in binary [30:55] by making a noise on your desk and have [30:57] the sound recorder convert that into [30:58] code, you do. You go ahead. You'll be [31:00] very slow. It won't be great. If you [31:02] want to use an IDE, great. If you want [31:03] to use AI to help you, great. But [31:05] fundamentally, you're the one that is [31:06] shipping that. There's still the human [31:08] element of that. So, you have to own [31:10] that. To what level you want to [31:12] understand that is probably down to the [31:15] individual's confidence level on a bunch [31:16] of things. Um, [31:19] but again, it's like at the moment, we [31:21] can't have agents sign legal contracts. [31:23] We can't have can't page an agent at [31:25] 3:00 a.m. I know there's a few firms [31:27] trying to claim that we can do that. Um, [31:29] but we still need humans for sure in so [31:31] much stuff. that this is [31:33] >> and also more importantly who is doing [31:35] the learning who is feeding that back [31:36] into architectural strategy right like [31:38] >> there is a crucial role for humans to [31:40] play sure we might be moving a level up [31:42] of the abstraction stack but you cannot [31:44] remove humans entirely or you just [31:47] washing accountability [31:49] >> yeah yeah um I've actually got a blog [31:52] post half written exactly covering the [31:54] kind of the learning angle of that [31:55] because yeah um the experiential stuff [31:58] might be going away and we need to find [32:00] ways to educate people But yeah, I think [32:02] the ownership does still very much sit [32:04] with the humans, be that engineers or [32:06] people like me as a leader or whatever. [32:08] But yeah, [32:10] >> so now that we've established who should [32:12] be the buck stops with who is running [32:16] this experiment even if AI is [32:19] implementing it. So how do you safely [32:23] experiment in this time period? What is [32:27] necessary for that? [32:32] I think that um [32:36] I think safe experimentation [32:40] um is an understanding of the audience. [32:43] I think Liz kind of pointed it out well. [32:45] It's like we're all human, right? When [32:47] you run an experiment, you're [32:48] interacting with people and you have to [32:50] take that into consideration. Um I think [32:54] a a safe experimentation program uses [32:58] some trusted methods, right? True [32:59] randomization. [33:01] Um it could potentially do isolation of [33:04] experiments, you know, keep out, you [33:06] know, control groups, you know, from [33:08] being exposed and using that as a [33:10] baseline constantly to to make sure that [33:14] you know that um you are [33:18] preserving [33:19] um you know your users access into the [33:22] system and and and testing in a you know [33:25] a responsible way. But I I I don't know [33:28] that anything in AI [33:31] changes that shift from an application [33:34] development perspective. That's my my [33:36] thought process anyways is that [33:39] AI is it's software at the end of the [33:42] day. It has end users at the end of the [33:44] day. We have to be responsible to our [33:46] end users. Um we have to be logical in [33:49] what we're testing and you know are we [33:51] creating an opportunity to learn [33:52] something given the parameters of the [33:55] test and you know what our hypothesis [33:57] might be. Uh and if you and if you're [33:59] confident you know you can say yes then [34:02] you've got a safe test on your hands. [34:06] I think much like the ownership angle [34:08] there the [34:11] the person working in growth be that a [34:12] PM or a growth person or even the [34:14] engineer they can't abdicate the [34:16] responsibility around [34:19] how you would design and then roll out [34:22] the experiment much like they've done in [34:23] the past um that still holds [34:28] >> yeah I would say responsible [34:30] experimentation requires that you have a [34:34] well functioning release process. It [34:36] requires that you have feature flags. It [34:38] requires you have observability, right? [34:39] Like there is a certain level of [34:40] operational table stakes maturity that [34:42] is required because if you are just [34:44] flipping flags, right? Like and you [34:46] can't actually measure the results like [34:48] you are not actually experimenting. [34:50] You're just injecting chaos, right? [34:52] There's a reason we talk about chaos [34:54] engineering, right? And I think that [34:56] when we talk about progressive delivery, [34:57] right? Like you have to actually deliver [34:58] something and validate it. H [35:01] >> it's like having a light switch in this [35:03] room controlling a light in some other [35:05] room and you're just flipping it on and [35:07] off and you have no idea you're causing [35:09] chaos. [35:12] >> Yeah, absolutely. Uh we did have a [35:15] question from the audience. So there [35:19] it's a very broad one. So the idea of [35:21] how do you quickly and safely recover [35:23] from incidents after a release when many [35:27] subsequent releases or deployments have [35:30] happened since. So this is definitely a [35:33] smaller releases traceability [35:35] observability. What else can help this [35:38] anonymous attendee thank you for your [35:40] question [35:42] do this quicker safer recovery? [35:49] I I I think if I understand, you know, I [35:52] I think what we have to know is what's [35:55] the source of the problem. Uh I think [35:58] with if you don't have certainty about [36:00] that, then you tend to roll back a lot [36:03] of things in the hopes that you capture [36:06] you know the the the pro the underlying [36:08] problem. [36:11] My belief is that if you are working in [36:14] smaller and smaller units and putting [36:16] those units behind code units and [36:18] putting those things behind flags, [36:21] you have a greater chance to better [36:24] understand if a specific change is [36:27] causing that problem. In which case you [36:28] can, it's a term that we use in the [36:32] industry, you can kill that flag. You [36:35] can turn off that negatively impacting [36:38] experience while leaving everything else [36:41] in place. If you don't work in those [36:44] small units and you don't work behind [36:46] flags and an incident is discovered and [36:49] there are many releases between now and [36:52] the incident, what can you do? You can [36:54] try to triage and obviously try to get [36:56] to the root cause. That means that pain [36:59] will continue. It exists until you can [37:01] figure that out. You can do a general [37:04] roll back strategy which might cause its [37:07] own friction because people were [37:08] enjoying the use of something that [37:10] suddenly might get taken away because [37:12] we're not sure if that was causing a [37:14] negative consequence. [37:16] And so you kind of have to, you know, I [37:21] would say you have to understand the [37:24] changes. You have to control those [37:25] individual changes more to be able to [37:30] not get into that cycle of, you know, [37:33] rolling everything back. [37:36] >> Certainly as a reliability practice, if [37:38] you are discovering issues relatively [37:40] quickly after they happen, the roll back [37:42] is the safest way to get into a known [37:44] good state. Um you know we know that [37:46] fixing forward can introduce all kinds [37:48] of complications. Um [37:51] but the argument is when you have [37:53] feature flags um you can revert to a [37:56] known state without changing any code [37:58] just uh changing the feature flags. And [38:00] I think in that state where you have had [38:03] a number of deploys and an issue was [38:05] latent and then it surfaced, right? Like [38:06] you know, yes, a fixed forward may be [38:09] your best option, right? But a lot then [38:12] goes into resilience strategies, right? [38:14] Like how are you making sure that it's [38:15] only smaller blast radius of users who [38:18] are impacted, right? Like how are you [38:19] quantifying who those users are? Are you [38:20] able to communicate with them? Right? [38:22] There's so many things that go into a [38:24] user's perception of how bad an outage [38:27] was um that are not strictly related to [38:29] the duration of the outage. [38:32] >> It could be the impact, it could be [38:33] financial, could be just how much it [38:35] annoyed their day if they're the [38:37] individual contributor that's using that [38:39] tool. Uh all these matter and something [38:44] we've been talking about the impact and [38:46] speed of AI and just there's more of [38:49] everything. But I think this is going to [38:51] be the year where we're not just hearing [38:52] we're going to hear about agents all the [38:54] time, but I think we're going to hear [38:55] about them also in the ops world because [38:58] this agentic operations [39:01] agentic is interesting because maybe a [39:06] specific skill can be trained around [39:09] this this threat posture that you have [39:11] that if you reach a certain threshold or [39:14] something, if that CPU spiked up like [39:16] you said, Liz, maybe It's something that [39:21] the AI will do something or at least [39:23] help you with your root cause analysis [39:25] because it can process more information. [39:28] Buck stops with that developer that's in [39:31] charge of the AI. But still maybe, [39:34] >> right? Like I think this is one of the [39:35] really cool and exciting things that [39:37] we're starting to see our customers do [39:39] with AI that we're doing ourselves. An [39:41] AI doesn't care about being wrong. It [39:44] can investigate six different parallel [39:46] strands, right? Like whereas a human can [39:48] probably only investigate one or two [39:50] investigative hypotheses at the same [39:51] time. Right? At the end of the day, [39:54] someone does have to make a decision [39:55] about what we are going to do about it [39:57] and to feed that learning back into our [39:58] software development process. But as far [40:00] as being able to find a bug quickly that [40:03] is starting to become faster for a [40:06] machine to do than for a human to do, [40:07] which is tremendously exciting. Mhm. [40:10] >> I was going to build on that [40:13] >> both the fact like kind of Liz said that [40:16] if you've done multiple releases and not [40:18] found the issue because I mean when we [40:20] did big rollouts of of things you tend [40:22] to be watching dashboards like a hawk [40:23] and looking for any sign that [40:24] something's not quite right. So you can [40:26] spot that early and roll back quickly [40:27] and then figure out the wise and [40:29] wherefors um on that one. Um, but to [40:33] that point, the other thought that [40:34] popped into my head was I remember [40:36] again, this is a GitHub memory, but when [40:39] things went wrong, we did a lot of git [40:40] bisect and things to try and work out [40:42] what might have changed that causes [40:43] things. And one of the really lovely [40:46] things for me uh about AI tooling is [40:49] that having joined a company relatively [40:50] recently, I don't have to go through all [40:52] of the git biceps and do git logs and [40:54] try and work stuff out. I ask the AI to [40:56] figure this stuff out for me in the [40:57] codebase. Um, and that's way faster and [40:59] to that point parallels. So we can we [41:03] can help ourselves uh we can have it [41:04] help us and and and do that [41:06] amplification that you mentioned right [41:08] at the start. [41:10] >> Just to take one last question from the [41:12] audience and it's kind of a big question [41:14] but I think we can make it a little [41:17] little and I think Ryan you kick off [41:19] with this. Uh Rajie thank you for [41:21] asking. How would you design a formal [41:24] life cycle and architectural strategy to [41:26] ensure that feature flags remain a [41:28] temporary scaffolding rather than a [41:31] permanent complexity? And how do you [41:33] enforce the removal of these flags once [41:35] a feature is 100% rolled out which [41:39] that's very you said Ryan you alluded to [41:41] that you were testing AI for this now [41:44] because it can be like hey we think this [41:47] is old which we love to stop zombie [41:50] things from running environmentally and [41:52] cloud cost but go more Orion. Yeah, it [41:55] there's quite a few things in in that [41:57] question, right? The first off, I I [42:00] think you're getting to the difference [42:02] between having a feature flagging system [42:05] and having a feature management system. [42:06] I think a feature flagging system is can [42:10] be implemented in a variety of ways and [42:12] you hear it you know whether uh folks [42:15] are doing config files or whether they [42:16] have database toggles or a variety of [42:19] other mechanisms that create some code [42:21] isolation and the ability to then change [42:25] um you know the behavior of that code. [42:27] Feature management in my definition [42:30] means that you know flags do have a life [42:32] cycle. There are flag types. There are [42:35] long running flags, just to be clear, [42:36] right? There are often shortlived flags [42:39] to support feature releases, but there [42:41] are long live patterns that make a lot [42:43] of sense. Um, [42:46] >> for sure, right? Like, you know, when [42:47] we're talking about the difference, [42:48] there's the difference between I have [42:50] this code that I want to enable and [42:51] disable. I'm doing a migration from path [42:53] A to path B, right? Like, that's clearly [42:54] a shortlived flag. [42:56] >> You should be notified. You should have [42:57] your agent rip out the old path when [42:59] it's no longer needed. But also we have [43:01] things like you know uh we call them [43:03] control control rods like a nuclear [43:05] reactor right like you know hey we need [43:07] to emergency drop load right like you [43:09] know that's not something that you want [43:10] to take out of your codebase or here's [43:12] the configuration for how many uh [43:14] segments we process in parallel per [43:16] worker right like again that's something [43:17] that you might want to runtime tune [43:20] >> knowing which one is which having that [43:22] taxonomy having the hooks into your [43:26] system to make sure that they are you [43:28] know that they're able to control the [43:30] roll out of the flag and also to receive [43:32] the signal that the roll out is done and [43:33] to rip out the code, right? Like that's [43:35] kind of that direction that's going. [43:37] >> Yeah. So once you have that ability to [43:39] define flag types that your [43:41] organizations are using, then you can [43:43] define that life cycle around them, you [43:45] know, how long should they should exist, [43:47] where should they persist, what's the [43:49] naming, what's something as simple as [43:51] the naming convention of different types [43:54] of flags to better understand their [43:56] purpose uh inside my code. And so you [43:59] want the ability to interrogate your [44:01] system to understand what's operating in [44:04] any given time inside your application [44:07] and what's the state of that flag once [44:10] it's discovered because an inventory of [44:12] flags gives you nothing right it doesn't [44:14] say if I have a 100 flags or a thousand [44:17] flags I can't say whether I have a [44:19] healthy implementation um it's what are [44:22] those flags doing should they be there [44:24] and if they don't have a purpose to the [44:26] last part of your question How do we [44:28] remove them? Now, I've heard or I've [44:30] experienced a variety of mechanisms in [44:33] terms of preventing that. I have [44:35] customers that actually prevent [44:37] developers from uh implementing new [44:41] changes until they retire flags, [44:44] right? They prevent actual pull requests [44:46] from going through. It's pretty [44:48] draconian. Um it's kind of hard-handed, [44:50] but you know, that's that's a symptom. [44:52] We're obviously trying to find nicer [44:55] mechanisms, things to support the [44:56] engineer in their effort of managing [44:58] this because there's a lot of cognitive [44:59] load associated with using flags. And so [45:02] AI for us is uh you know for us uh you [45:06] know what we think is is going to help [45:08] an engineer the most in managing that [45:11] life cycle and just saying hey kind of [45:13] seems like this is no longer in use. [45:16] You're the human in the loop. Can we [45:18] remove it uh and take some action on [45:21] your behalf? [45:22] I think one other interesting piece that [45:24] you discussed was the kind of validation [45:27] and the knowing what the landscape of [45:28] what's out there right if you are not [45:31] attaching the state of each flag to each [45:33] request right like then you are not [45:35] getting valuable telemetry data about [45:37] you know hey this outage might be [45:39] affecting only users with this [45:40] particular flag set right like so that [45:42] operational data is really really [45:44] important to feed into your processes [45:50] >> we can talk about this all day. I think [45:53] if we're going to end and wrap on a [45:55] note, [45:57] all of these we're talking about [45:59] technical solutions to a sociotechnical [46:01] problem. We all see great value in the [46:05] developers, the operators, the humans in [46:09] the tech processes and everything we're [46:12] talking about is hopefully making your [46:15] lives a little easier because that was [46:17] supposed to be the idea of AI supposed [46:20] to let us work on unique, interesting, [46:23] novel problems. So hopefully we wish [46:27] that to you and with the AI of things or [46:30] not, progressive delivery is one of [46:33] those best practices that will help you [46:35] succeed, get fast feedback, and be [46:39] better at your work. So we wish you luck [46:42] and we'll see you in the lead dev slack [46:45] if you need anything and we'll see you [46:46] at the next one. Have a great day, [46:48] y'all. Thank you so much, Liz, Claire, [46:51] and Ryan. Thank you. [46:53] >> Thank you. Bye everyone.  📄 **Full text:** ![[Full Document Contents/Articles/Derisking Releases in the AI Era|Derisking Releases in the AI Era, YouTube]] ## Highlights - honestly, I don't think AI changes the things because if you were doing the DevOps and engineering best practices and guardrails that we've been talking about for years, AI, you should be logically able to keep up with this new pace. You probably also aren't releasing as much. You are not adding that many lines of code because your quality checks are in place and things. But it is if you don't have those in place and ([View Highlight](https://read.readwise.io/read/01kmjrp9j59s83r5rga63qhfga)) - Tags: 👻 ai highlighted - the infrastructure capability to do progressive rollouts. This is like back when I started with gold CDs being pressed for your master version of the software that you released. Just dating myself slightly >> Ryan. Yeah. What do you want to add to that on the psychological safety? Oh yeah. Yeah. You know, um, you know, progressive delivery is a is a is a very interesting topic to me. I've been preaching it for a very long time. Um, specifically around using obviously, you ([View Highlight](https://read.readwise.io/read/01kmjrp9k43c10mp2n31brrza8)) - Tags: 👻 ai highlighted - right? Like no technological change is actually going to make a difference if your team is afraid to interact with production in any way because they, you know, it pages them all the time or they get yelled at every time they cause an outage. >> So we have feature flags. Maybe we want to quick to find that in like a few words and then let's try to run popcorn. >> I think we should back up a step though, right? like you know when CLA was talking earlier about this idea of um ([View Highlight](https://read.readwise.io/read/01kmjrp9m8ajmxp93kmf4sdh2w)) - Tags: 👻 ai highlighted - which um to Liz's point yeah that does spot a lot of things before they hit the wider audiences. I think one interesting AI tiein here is that we've talked a lot about how AI puts stress on the system, >> but I also see AI as an opportunity to make it easier to do progressive delivery in a lot of ways, you know, >> right? Like development teams struggle when you tell them, you know, hey, can you write like, you know, can you put a feature flag around this? And they're like, I don't know how to do this, right? Like ([View Highlight](https://read.readwise.io/read/01kmjrp9p4nheh9eead5v3ansp)) - Tags: 👻 ai highlighted - interrogate your flags and say, are you worthy of still existing? Right? And if not, we're going to we're going to do a pull request uh and we're going to suggest that you get removed. Right? So all all stages of the life cycle can be affected by AI as it relates to flag management in our opinion. >> Also, what about other things? We're talking about feature flags, but there's so much more to progressive delivery. What about has anyone used blue green or AB testing ([View Highlight](https://read.readwise.io/read/01kmjrp9srsv9v0app89v7qpxj)) - Tags: 👻 ai highlighted - out how to prioritize, right? Like doing these AB tests and not testing every feature on, you know, the same set of unsuspecting users because they will get very unhappy when you keep turnurning their UI. >> Yeah, there's an interesting tension there for me with the AI side because you've got the the constantly changing churn and yes, I've had enterprise customers at previous jobs say, "Why is the interface changing?" And you're like, "It's only changed once in a year." And they're like, "No, not happy." Um, but we're also like there's the hyper localized hyperpersonal software that's being built at speed and like where does this all be play out ([View Highlight](https://read.readwise.io/read/01kmjrp9z9pnp531gvdvj64wga)) - Tags: 👻 ai highlighted - experimentation requires that you have a well functioning release process. It requires that you have feature flags. It requires you have observability, right? Like there is a certain level of operational table stakes maturity that is required because if you are just flipping flags, right? Like and you can't actually measure the results like you are not actually experimenting. You're just injecting chaos, right? There's a reason we talk about chaos engineering, right? And I think that when we talk about progressive delivery, right? Like you have to actually deliver something and validate it. H ([View Highlight](https://read.readwise.io/read/01kmjrpa9z1c8j6wjraygy0qcm)) - Tags: 👻 ai highlighted - as being able to find a bug quickly that is starting to become faster for a machine to do than for a human to do, which is tremendously exciting. Mhm. >> I was going to build on that >> both the fact like kind of Liz said that if you've done multiple releases and not found the issue because I mean when we did big rollouts of of things you tend to be watching dashboards like a hawk and looking for any sign that something's not quite right. So you can spot that early and roll back quickly and then figure out the wise and wherefors um on that one. Um, but to ([View Highlight](https://read.readwise.io/read/01kmjrpapbyre3vhe2j4bnt8gr)) - Tags: 👻 ai highlighted