How to build the guardrails your AI releases actually need

How to build the guardrails your AI releases actually need ![rw-book-cover](https://img.youtube.com/vi/Xt7y6CKDqcI/maxresdefault.jpg) 🔗 **Source:** [How to build the guardrails your AI releases actually need](https://www.youtube.com/watch?v=Xt7y6CKDqcI) > [!note]+ Document Notes > tags: [ai-deployment, feature-flags, continuous-delivery, guardrails, software-validation] > author: Eric Nukem > source: Harness > reference: https://www.youtube.com/watch?v=Xt7y6CKDqcI > date created: {{date:YYYY-MM-DD}} {{time:HH:mm}} > > # 🏷️ Title: How to build the guardrails your AI releases actually need >  ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ## 📝 Summary - **What happened**: AI coding tools are increasing feature velocity 4-20x, but organizations lack proper guardrails for testing and deployment phases. - **Why it matters**: Without proper controls, AI-generated code creates security vulnerabilities, quality issues, and deployment bottlenecks despite faster development. - **Key stakeholders**: Development teams, DevOps engineers, security teams, product managers, experimentation teams. - **Risks and tradeoffs**: Faster coding vs. increased deployment risk; automation efficiency vs. human oversight needs; feature velocity vs. technical debt accumulation. - **Bottom line**: Success requires combining continuous delivery with feature flags, implementing three types of guardrails (human judgment, security evidence, policy enforcement), and validating releases through experimentation. ## 📖 Main Content The presentation addresses a critical gap in AI-assisted software development: while organizations excel at providing AI coding tools, they struggle with the downstream effects of dramatically increased feature velocity. Eric Nukem from Harness identifies that teams are seeing 4-20x more features in deployments but lack adequate systems for safe release management. ### The AI Code Velocity Problem Current AI coding adoption shows strong uptake in development phases—developers use Claude, Copilot, Cursor, and ChatGPT effectively. However, survey data reveals concerning gaps: - Most organizations still require human review of every line of AI-generated code - Less than half have implemented AI guardrails - Only 36% report AI improving software quality - 58% are running more experiments but systems can't handle the volume ### Solution Framework: Guardrails + Validation The presentation proposes a two-pronged approach focusing on testing/deployment and release/experimentation phases: **Three Types of Guardrails:** 1. **Human judgment**: Approvals for production, high-risk services, unusual deployment patterns, failed verification scenarios 2. **Security evidence**: Application security testing with vulnerability deduplication and remediation workflows 3. **Policy enforcement**: OPA-style policies ensuring naming conventions, compliance standards, and organizational consistency **Validation Process:** - Release process integration with feature flags - Progressive rollout capabilities (1% → segments → full deployment) - Experimentation on every feature release - Technical debt cleanup through automated feature flag removal ### Continuous Delivery + Feature Flags Integration The core technical approach combines traditional CD deployment with granular feature flag control. While CD handles application-level deployments, feature flags enable individual feature rollouts with instant kill switches. This reduces blast radius from full rollbacks affecting all features to targeted flag toggles affecting only problematic features. ## 🗒️ Notes #### ⏱️ [00:03:12](https://www.youtube.com/watch?v=Xt7y6CKDqcI?t=192) AI Development Workflow Problems - Coding and review phases well-supported with AI tools - Deployment size explosion: 4-20x more features per release - Testing and deployment become bottlenecks - Release and experimentation validation critical for AI-generated features #### ⏱️ [00:08:30](https://www.youtube.com/watch?v=Xt7y6CKDqcI?t=510) Survey Results on AI Adoption - Human review still required for every line of AI code - <50% have AI guardrails implemented - 58% running more experiments but systems can't scale - Only 36% see quality improvements from AI #### ⏱️ [00:15:45](https://www.youtube.com/watch?v=Xt7y6CKDqcI?t=945) Continuous Delivery vs Feature Flags - CD: Build → Test → Deploy as single unit with canary rollouts - Problem: Rollback affects all features in deployment - Feature flags: Individual feature control with progressive rollout - Best practice: Combine both for deployment flexibility and risk mitigation #### ⏱️ [00:22:10](https://www.youtube.com/watch?v=Xt7y6CKDqcI?t=1330) Three Guardrail Types - Human judgment: Production approvals, high-risk services, unusual patterns - Security evidence: Vulnerability scanning, deduplication, remediation - Policy enforcement: OPA policies for naming conventions and compliance #### ⏱️ [00:28:15](https://www.youtube.com/watch?v=Xt7y6CKDqcI?t=1695) Validation Framework - Release process integration prevents bottlenecks - Progressive rollout: Internal users → segments → full deployment - Experimentation requirement on all features - Technical debt cleanup through automated flag removal #### ⏱️ [00:35:20](https://www.youtube.com/watch?v=Xt7y6CKDqcI?t=2120) Speedway Motors Case Study - Before: 2 releases/month, no conversion rate visibility - After: Multiple daily releases, immediate ROI measurement - Integration enabled rapid iteration and business impact validation ## 📚 References - [Harness Flagship 2026 Event](https://www.harness.io/flagship) - [D-risking Releases in the AI Era Report](https://www.harness.io/resources) - [Speedway Motors Implementation Case Study](https://www.harness.io/customers/speedway-motors) - [Harness Product Demo](https://www.harness.io/demo) ## 🗣️ Paraphrase Organizations are experiencing a paradox with AI-assisted development: while coding velocity has increased dramatically, the benefits aren't reaching production due to inadequate release management systems. The solution isn't to slow down development but to implement proper guardrails and validation processes that can handle the increased throughput. The key insight is that traditional deployment approaches—treating all features as a monolithic release—become untenable when AI generates 20x more features. Feature flags provide the granular control needed to manage this complexity, allowing teams to validate individual features without risking entire deployments. However, this requires integrated tooling that spans the entire pipeline from development to production. ## 📰 Newsletter **AI Code Velocity: The Hidden Deployment Crisis** Your developers are shipping code faster than ever with AI assistance, but are those features actually reaching users? A new Harness survey reveals that while AI coding tools see widespread adoption, most organizations lack the guardrails needed for safe, rapid deployment. The problem: AI-generated features create 4-20x larger deployments, but traditional rollback strategies affect all features when problems arise. The solution combines continuous delivery with feature flags, enabling granular control and instant kill switches for individual features. Key takeaways for WordPress professionals: - Implement progressive rollout strategies (1% → segments → full) - Require experimentation on every feature release - Automate technical debt cleanup to prevent AI-generated cruft - Integrate security scanning into deployment pipelines Tools to explore: Feature flag services, experimentation platforms, policy-as-code frameworks, automated cleanup utilities. ## ✅ Action Items #### DevRel - Create educational content on feature flag implementation patterns for AI-generated code - Develop workshops on progressive rollout strategies and blast radius reduction - Build demo environments showing CD + feature flag integration - Document security scanning integration patterns for AI code validation #### OSPO - Establish policies for AI-generated code review requirements in contribution guidelines - Implement security scanning requirements for AI-assisted contributions - Create governance framework for feature flag naming conventions across projects - Define SLAs for AI code vulnerability remediation processes #### WordPress - **Core**: Evaluate feature flag integration patterns for core feature releases - **Core Performance**: Implement progressive rollout for performance optimizations - **Security**: Establish AI code scanning requirements for security-sensitive areas - **Community**: Create guidelines for plugin developers using AI coding tools - **Hosting**: Develop recommendations for hosts managing AI-generated plugin deployments #### The WPCC - Fund research into WordPress-specific feature flag implementations - Support development of AI guardrail templates for WordPress projects - Create grants for experimentation platform integration with WordPress workflows ## 🧪 Hypotheses 1. **Integration Hypothesis**: Organizations using integrated CD + feature flag platforms will achieve 2-5x faster time-to-production for AI-generated features compared to those using separate tools. 2. **Guardrail Effectiveness**: Teams implementing all three guardrail types (human judgment, security evidence, policy enforcement) will reduce AI-related production incidents by 70-90% within 6 months. 3. **Experimentation Scaling**: Feature flag integration enables experimentation teams to scale from weekly to daily experiment launches without increasing team size. 4. **Technical Debt Prevention**: Automated feature flag cleanup reduces AI-generated technical debt accumulation by 60% compared to manual cleanup processes. 5. **Progressive Rollout Impact**: 1% → segment → full rollout patterns reduce blast radius of AI code issues by 95% compared to full deployment strategies. ### 📊 Methodology - A/B testing comparing integrated vs. separate toolchain outcomes - Longitudinal studies tracking incident rates before/after guardrail implementation - Survey research on experimentation team capacity and throughput - Code quality metrics analysis for cleanup automation effectiveness - Case studies measuring deployment risk reduction through progressive rollout ### ❓ Questions - How do feature flag naming conventions scale across large organizations with thousands of developers? - What are the optimal progressive rollout percentages for different types of features (UI, API, infrastructure)? - How can teams balance automation with human oversight without creating approval bottlenecks? - What security scanning tools are most effective at catching AI-generated vulnerabilities? - How do you measure the ROI of guardrail implementation versus deployment velocity? ### 🔮 Future Research - AI-assisted feature flag management and automated rollout decisions - Integration patterns between WordPress-specific deployment tools and feature flag platforms - Security vulnerability patterns specific to AI-generated WordPress plugins and themes - Cost-benefit analysis frameworks for guardrail implementation - Machine learning approaches to predicting optimal rollout strategies ### 💡 Implications - **Practice**: Development teams need to expand focus beyond coding to include deployment pipeline optimization - **Policy**: Organizations require new governance frameworks specifically addressing AI-generated code risks - **Security**: Traditional security review processes may be inadequate for AI code volume and unique vulnerability patterns - **Education**: Developer training must include feature flag management and progressive rollout strategies - **Funding**: Investment in integrated toolchains provides better ROI than point solutions for AI development workflows ### 🧠 Learning Styles Assessment - **Reading level estimate**: Flesch-Kincaid Grade Level ~12 (technical business presentation) #### 👀 Visual learners - **Present**: Architecture diagrams showing CD + feature flag integration, pipeline flow charts, deployment percentage visualizations - **Improve**: Add before/after deployment risk diagrams, feature flag decision trees, security scanning workflow visuals #### 🔊 Aural learners - **Present**: Webinar format with presenter narration and Q&A segments - **Improve**: Provide audio-only version, add discussion group integration, create podcast-style follow-up content #### 🗣️ Verbal learners - **Present**: Clear definitions of technical terms, analogies (banking app example), structured explanations - **Improve**: Add glossary of terms, provide story-driven case studies, include plain-language summaries #### 🧩 Kinesthetic learners - **Present**: Demo offers, hands-on tool exploration suggestions, QR codes for immediate action - **Improve**: Create guided lab exercises, provide configuration templates, offer sandbox environments #### 🧠 Logical learners - **Present**: Structured framework (guardrails + validation), survey data, measurable outcomes - **Improve**: Add decision matrices, ROI calculations, step-by-step implementation checklists #### 🧑‍🤝‍🧑 Social learners - **Present**: Q&A interaction, community event promotion, peer examples (Speedway Motors) - **Improve**: Include discussion forums, peer review components, collaborative implementation guides #### 🧘‍♀️ Solitary learners - **Present**: Self-paced webinar format, downloadable resources, individual demo scheduling - **Improve**: Add self-assessment tools, provide reflection prompts, create independent study materials ### 📊 Accessibility Assessment #### Screen reader semantics - **Present**: Structured presentation with clear headings and sections - **Improve**: Add detailed alt text for diagrams, ensure slide content is screen reader accessible #### Keyboard navigation - **Present**: Standard webinar platform navigation - **Improve**: Verify all interactive elements are keyboard accessible, provide navigation shortcuts #### Contrast and readability - **Present**: Professional slide design with adequate contrast - **Improve**: Verify WCAG 2.2 AA compliance for all visual elements, test with high contrast modes #### Media and captions - **Present**: Live webinar with Q&A capability - **Improve**: Provide auto-generated captions, offer transcript downloads, ensure audio quality ### 🌍 DEIB Assessment #### Cultural breadth - **Present**: Focus on technical practices applicable across organizations - **Improve**: Include examples from diverse company sizes and industries, consider global deployment patterns #### Inclusive language - **Present**: Professional, technical language appropriate for business audience - **Improve**: Define technical jargon consistently, provide alternative explanations for complex concepts #### Neurodiversity support - **Present**: Structured presentation with clear sections and predictable flow - **Improve**: Provide content outline upfront, use consistent formatting, offer multiple content formats #### Representation - **Present**: Business-focused content with practical examples - **Improve**: Include diverse organizational examples, acknowledge varying team structures and capabilities ### ✅ Logical Validity #### Strong reasoning - **Deductive reasoning**: Clear logical progression from AI code velocity increase → deployment complexity → need for guardrails - **Inductive reasoning**: Survey data supporting conclusions about industry-wide challenges - **Causal reasoning**: Direct connection between toolchain integration and improved outcomes demonstrated through case study #### Fallacies - **Selection bias**: Speedway Motors case study may not be representative of all organizations - **Solution bias**: Presentation heavily favors Harness-specific toolchain without adequate comparison to alternatives - **False dichotomy**: Implies organizations must choose between speed and safety rather than exploring other approaches #### Tone of voice - **Professional and authoritative**: Uses industry terminology and data to establish credibility - **Solution-oriented**: Focuses on practical implementation rather than theoretical concepts - **Vendor-neutral framing with product integration**: Acknowledges multiple tools while showcasing Harness capabilities ### 🛡️ DARVO Analysis - **Deny**: No denial patterns present - speaker acknowledges industry-wide challenges directly - **Attack**: No attacks on critics or alternative approaches - maintains professional tone throughout - **Reverse victim/offender**: No victim/offender reversal detected - presents collaborative problem-solving approach ## 🎯 Conclusion AI-generated code velocity creates a new category of deployment risk that traditional release management approaches cannot adequately address. The solution requires integrated toolchains combining continuous delivery with feature flags, supported by comprehensive guardrails spanning human judgment, security validation, and policy enforcement. Organizations that successfully implement these practices will realize the full potential of AI coding acceleration while maintaining system reliability and security. The key is recognizing that faster coding requires proportionally more sophisticated deployment and validation processes, not just scaling existing manual approaches. ## 🔄 Refactor Appendix — How to build the guardrails your AI releases actually need > Density: 2–5 Literature atoms (H3), 0–3 Permanent candidates (H4). > Defaults: 3 Literature, 1 Permanent. Resonance filter: if you can't > articulate in one line why a claim matters, drop it. ### 📚 Literature Atoms (H3) #### AI code velocity creates deployment bottlenecks without proper guardrails > [!quote] ID:rw_ai_velocity_001 > We're seeing four to 20 times the number of changes being pushed out the door or at least through the process that we were seeing before. And we all know what happens when we have an increased number of things changing at any given time. That opens up the risk that something's going to go wrong. **Context & use**: - Core problem statement around 8:30 in presentation - Why this matters: Identifies the fundamental mismatch between AI coding speed and deployment capability **Tags**: #xref #articleseed **Highlight**: [view highlight](https://www.youtube.com/watch?v=Xt7y6CKDqcI?t=510)  #### Feature flags enable granular rollback without affecting entire deployments > [!quote] ID:rw_feature_flags_002 > If you have a similar problem where you've rolled out a small feature, one of the small features in your deployment to your power users or your premium users and they start having a problem, you can turn that flag off without having to do a roll back. You can just turn off that individual flag. **Context & use**: - Technical explanation around 15:45 showing CD vs feature flag approaches - Why this matters: Solves the blast radius problem when AI generates many features per deployment **Tags**: #xref **Highlight**: [view highlight](https://www.youtube.com/watch?v=Xt7y6CKDqcI?t=945)  #### Three guardrail types required for AI code validation > [!quote] ID:rw_guardrails_003 > What are three types of guardrails we can put up? Well, one, we can put up human judgment. Also, we need some sort of security evidence. And then you need policy enforcement because AI is really good about coming up with its own workarounds for things. **Context & use**: - Framework presentation around 22:10 defining comprehensive guardrail strategy - Why this matters: Provides actionable taxonomy for implementing AI code safety measures **Tags**: #articleseed #xref **Highlight**: [view highlight](https://www.youtube.com/watch?v=Xt7y6CKDqcI?t=1330)  ### 🧠 Permanent Candidates (H4) #### AI-generated feature velocity requires proportional deployment sophistication **Claim** (one sentence, Courtney's voice): Organizations achieving 4-20x coding velocity through AI must implement equally sophisticated deployment and validation processes to realize benefits safely. **Evidence pointer**: [[#AI code velocity creates deployment bottlenecks without proper guardrails]] **Scope & conditions**: Applies to teams using AI coding tools extensively; breaks down when deployment infrastructure cannot scale with development output **Links**: - [[DevOps transformation patterns]] — because extends existing CD practices - [[Technical debt management]] — because applies-in high-velocity development contexts - [[Risk management frameworks]] — because contrasts with traditional deployment risk models  ### ✏️ Fleeting Queue - Progressive rollout percentages (1% → segments → full) need optimization research for different feature types #fleeting #inbox - Integration patterns between WordPress deployment tools and feature flag platforms worth exploring #fleeting #inbox - Policy-as-code examples for AI coding standards could be valuable community resource #fleeting #inbox ### → Things 3 (max 2 actions) - [ ] Research feature flag integration options for WordPress core development workflow #things - [ ] Evaluate automated cleanup tools for managing technical debt in AI-assisted projects #things --- ## Full Transcript [00:12] Well, hey everybody. Welcome. We are [00:14] going to talk about how to build the [00:16] guard rails your AI releases actually [00:19] need. Welcome and thanks for joining. My [00:21] name is Eric Nukem. I'll do a little [00:23] intro in a minute, but uh please note [00:24] that we will have be taking questions [00:27] throughout. So if you have a question [00:30] comes up randomly at a given section, [00:32] you know, don't hold back. Go ahead and [00:34] open that question box in the tool and [00:37] uh go ahead and fill it out anytime you [00:39] want. Have it ready so you can be asking [00:41] those questions and I will get to those [00:43] at the end if I don't see them as I'm [00:44] going through here. So uh please do [00:47] that. Anyway, my name is Aaron Nukem. I [00:49] work for Harness. I work on our feature [00:51] management and experimentation uh [00:55] product. That product came from Split. [00:57] You might remember Split a while back [00:58] was acquired about a year a little over [01:00] a year ago was acquired by harness and [01:02] we've now integrated that into our [01:04] platform and I'm going to show you some [01:05] ways that that integration is uh luckily [01:08] helping to deal with this problem of the [01:11] AI coded uh feature velocity coming our [01:15] way. So let's go ahead and get right [01:17] into it and I'll tell you a little bit [01:19] more about uh what we're going to be [01:20] talking about today. So, I've done, you [01:22] know, kind of the introduction here. Um, [01:24] but we're going to talk about some AI [01:26] implications. I think this will resonate [01:28] with most people. Every customer I talk [01:30] to is dealing with this problem. So, I [01:32] think this will be really good for uh [01:34] for us to talk about today. And then [01:36] I'll give you a little bit of an [01:38] overview of harness. Not really going [01:39] into too much detail, but I just Harness [01:41] does so much and I want to make sure [01:43] we're focused on just the elements that [01:45] I'm going to be uh focusing on today. Uh [01:49] I'll also give you a little bit of an [01:51] overview of uh continuous delivery and [01:53] feature flags, what the differences are, [01:55] how they come together, and then of [01:57] course how they come together to solve [01:59] this particular problem. We'll talk a [02:01] lot about guardrails and validation. I [02:03] don't want to give that one away too [02:04] much, but those are the two areas we're [02:06] going to focus on to kind of fix some of [02:08] the some of the issues we're having [02:09] right now due to AI generated code. And [02:12] then I'll give you some resources and of [02:14] course we'll at the end there'll be time [02:15] for a few more questions. So, like I [02:18] said, don't hesitate. Don't wait till [02:19] the end to ask them. Go ahead and ask [02:21] your questions as we go through here. [02:22] That's the best way to do it. And I [02:23] think we'll also have a poll here at [02:25] some point to try to see where everyone [02:26] is at on this spectrum of how they're [02:28] dealing with this problem, too. So, [02:30] let's just go ahead and get into it. So, [02:33] when I talk about uh you know, the [02:35] problems with AIdriven code workflows, [02:39] what am I talking about? Well, in [02:41] talking to customers and I think [02:43] industrywide, even if you just read the [02:46] news around what's going on with AI, I [02:48] think it's clear that people are okay [02:51] with the coding and some of the review [02:54] part of the process. So, people are [02:58] using Claude, people are using C-Pilot, [03:01] people are using cursor, they're using [03:02] chatbt. they can produce code quickly [03:06] and companies have been very willing to [03:08] give people the uh the tools that they [03:12] need. They're not holding back on any of [03:13] this, right? Uh in fact, most people are [03:17] encouraging people to use the tools that [03:20] they are uh uh that they that they want, [03:23] right? They're not it's it's like no, [03:25] you need to use more tokens. [03:27] Some people are being measured on that, [03:28] I think. So, the coding part is pretty [03:31] well taken care of. people are are um [03:34] really excited to use those tools. No [03:36] problem there. Now, what happens? We [03:38] start developing more and more code. We [03:40] start creating more and more features [03:41] that we want to release. Things are [03:43] going great. Things are looking great. [03:45] But then we get to the point where we [03:46] need to roll these things out. And what [03:47] happens to the size of our deployments? [03:49] They explode, right? So, you may have [03:51] had a deployment in the past with, I [03:53] don't know, five, 10 features or [03:55] something like that. We're seeing four [03:58] to 20 times the number of changes being [04:02] pushed out the door or at least through [04:04] the process that we were seeing before. [04:06] And we all know what happens when we we [04:08] we have an increased number of things [04:10] changing at any given time. That opens [04:12] up the risk that something's going to go [04:13] wrong. So, we have to start focusing on [04:16] testing and deployment a bit more. And [04:18] we can turn some of that over to AI. But [04:20] as I'll show you in a minute, there are [04:22] some easy ways to also mitigate some of [04:24] the problems that happen when you do [04:26] that. And then uh last but not least, we [04:28] have release and experimentation. And [04:31] this last part of the puzzle is where [04:33] you actually want to use something like [04:36] feature flags or an experimentation tool [04:38] to validate that those changes that you [04:41] have made with AI are actually doing [04:43] what they're supposed to be doing and [04:45] especially not causing a problem with [04:47] either a security vulnerability or a um [04:52] an impact to your user experience. [04:54] Right? So that seems to be where we we [04:57] want to concentrate on improving at this [05:00] point. And why do I say that? [05:02] Well, uh we actually did a survey [05:04] recently and we t we asked um real users [05:08] about how they're dealing with this [05:10] problem and we got back some really [05:12] interesting results and and hang on till [05:13] the end because I'm going to give you a [05:15] link that you can use to download this [05:17] particular report and read all the [05:19] findings, not just the highlighted ones [05:20] I've pulled out here. So in that report, [05:23] what did we find? Well, we found that, [05:25] you know, a lot of people still need [05:27] human review for every single line of AI [05:30] generated code. And if you're writing a [05:33] lot more code, but you still need a [05:35] human to review every line of it, that's [05:37] not very efficient. And it might look [05:39] like your your code development engine [05:41] is running great, but you're still not [05:43] getting those features out into the wild [05:45] um at the rate that that uh you would [05:47] like to. [05:49] Uh we also found that less than half of [05:52] people have put in uh AI guard rails for [05:56] AI generated code. Um this could be a [05:59] real problem. We've all seen stories in [06:00] the news where somebody didn't have a [06:03] guardrail in place and they deleted [06:04] their codebase or they deleted customer [06:07] orders or what have you and uh the AI [06:10] just went ahead and did it and you know [06:11] was like oh sorry you know my bad you [06:14] know and that's not good enough in this [06:15] day and age. Um now what we did see was [06:18] we saw that 58% of people are running [06:20] more experiments. This is a good thing [06:22] uh in one way which is they're trying to [06:24] keep up with um running more experiments [06:28] because they have more features. But the [06:30] problem is their systems may not be able [06:32] to really support all those extra [06:34] running experiments because they're not [06:37] efficient enough or they're not geared [06:38] to that volume. we really should be [06:40] running experiments on every feature [06:42] that we release or that should be the [06:44] exception is when we don't do it right. [06:47] Um, and last but not least, only 36% [06:49] said AI had improved the quality of [06:50] their software. Again, sort of similar [06:53] to the first point in that you need to [06:55] go back and actually review things or [06:57] the results aren't quite up to what you [06:59] thought they would be, right? You you [07:01] thought that the quality of software [07:02] would be great. Turns out there's bugs [07:04] that are getting into the mix and [07:06] hopefully you're catching those either [07:07] in test or behind a feature flag so you [07:09] can turn those off quickly. So, a lot of [07:11] really interesting results. Again, I'll [07:13] give you the link for this when we get [07:14] to the resources at the end or they'll [07:16] be in the resources of your uh when you [07:18] uh after the webinar when you get the [07:20] resources in your email. Um, okay. Well, [07:23] let's take a I'm really curious to know [07:25] how everyone on the call is doing with [07:28] this. And so let's just do a quick poll [07:32] and see what you're doing. It should [07:33] have popped up on your screen now. And [07:34] the poll says, "How mature is your [07:37] team's use of AI in the SDLC?" So go [07:40] ahead and fill out that poll. The [07:42] choices are, you know, you're not using [07:43] it all. Maybe you're using it for, you [07:45] know, a few people are experimenting [07:47] with AI. Maybe you're using it, but it's [07:50] supervised. Maybe AI is uh integrated [07:54] with guard rails. And maybe no humans [07:56] are involved anymore. So go ahead and [07:58] vote on those options and let me know [08:01] where you're at in your journey here, [08:04] your AI journey in deploying and [08:07] releasing new features with AI. [08:15] While we're doing that, I can tell you [08:16] that in talking to a few folks, uh we [08:19] found out that um some people have gone [08:22] a little too fast here. So, uh, I was [08:26] talking to one Fortune 500 company [08:28] recently and they they gave, you know, [08:31] everybody all the tools they need and [08:33] they didn't put in any guardrails and [08:35] what they found were that that people [08:36] were so used to using AI. I'm sure many [08:39] of you have felt have seen this. I I [08:41] have myself where you you start to trust [08:43] the AI maybe a little too much and they [08:45] had bugs creeping in and because the [08:47] demand was so high on the developers, [08:49] they weren't actually reviewing anything [08:51] and they were just hitting, "Yep, looks [08:52] okay. go ahead and uh do that build or [08:54] or commit that code and uh bugs were [08:57] creeping in so they had to go back to a [08:59] much more manual code review process. Um [09:03] and so we are seeing that as well. [09:07] Okay. Well, let's just go ahead and take [09:09] a look at the results of the poll. So, [09:12] you know, it looks like most everybody [09:14] is right in the middle here. Uh AI is [09:17] used but not supervised. So, people are [09:19] still struggling with this problem. you [09:21] know, they're they're trying to use AI [09:22] in the SDLC, but uh they're not a [09:25] actually able to to get all the way to, [09:28] you know, the Nirvana, I guess, if you [09:30] want to say, where no humans are [09:31] involved anymore. But yeah, about 43% of [09:34] you are right in the middle where you're [09:36] using AI, but still very much [09:38] supervised. Um, you know, a good portion [09:41] of you, about a little less than a third [09:43] are actually using AI with integrated [09:45] guardrails and then the rest are still [09:47] experimenting. So, [09:49] uh, very interesting and very much on [09:51] par with the other times that I've asked [09:53] these questions. [09:55] Okay, so let's go ahead and move on to [09:57] how do we solve the problem, right? [09:59] That's what the bulk of this is going to [10:00] be about. So, uh, we're not going to [10:02] focus on the coding problem because you [10:04] all are really good about giving tools [10:06] to your developers to code with. Let's [10:08] focus on the testing and deployment and [10:10] release and experimentation phases of [10:13] doing your uh um of releasing new [10:16] features, right? And so I thought we [10:18] would tackle this in two different ways, [10:20] right? For the testing and deployment [10:22] and release and experimentation uh [10:24] phases, we're going to be talking about [10:26] uh how to set guard rails, what kind of [10:28] guard rails are should be in place, and [10:30] then also how to validate when you've [10:33] actually released a feature. So [10:35] validation is super important here. I've [10:37] talked to uh some companies that that [10:40] basically have uh I won't say dictated [10:42] because they haven't quite gone that [10:44] far, but what they've said is the goal, [10:46] what they're measuring against is [10:47] whether they're running an experiment on [10:49] every single feature release. Super [10:52] important. And also as part of that [10:55] validation comes what I calling for now [10:58] codification or that last part of the [11:00] process where you need to go in and [11:02] clean up your code. So, because I work [11:04] with feature flags, I'm going to show [11:05] you an example of how you can go clean [11:07] up feature flag code as well. [11:11] All right, before I get into those two [11:13] things, I do want to just set the stage [11:15] here, uh, and tell you a little bit [11:17] about what I'm going to be using my, you [11:20] know, obviously I come from harness, so [11:22] I'm going to be talking about our tools [11:23] here. You may have other tools, that's [11:25] totally fine. Maybe you've got a DIY [11:26] approach, whatever applies to you, but I [11:29] just want to let you know my context for [11:30] this. So um Harness has built basically [11:34] an AI um foundation if you will with the [11:38] Gentic Workflows software delivery [11:41] knowledge graph in the middle that pulls [11:42] in information from different areas so [11:44] it has context and then also intelligent [11:47] workflow orchestration. Um and that [11:51] results in not just feature flags but a [11:54] bunch of different products that we [11:55] sell. Uh we have products for velocity, [11:58] we have products for reliability, we [12:00] have products for security, and we have [12:02] products for efficiency, including [12:04] things like cloud cost management. And [12:06] so we're not going to be talking about [12:07] all of this today. We're really going to [12:09] only be talking about continuous [12:10] delivery and feature feature management [12:13] and experimentation, although I'll throw [12:15] a little bit a little plug in there for [12:16] security, too, because we can never [12:17] forget that. But th those are the two [12:19] areas I'm going to be focusing on today. [12:24] All right. So, let's talk about a little [12:26] bit about continuous delivery and then [12:28] I'll talk a little bit about feature [12:29] flags in a minute. Um, but I want to [12:32] make sure that people understand the [12:33] difference uh between the two tools and [12:35] how they apply here, right? So, with [12:37] continuous delivery, I mean, that's [12:39] pretty much what people know harness [12:40] for. That's our bread and butter. That's [12:42] where we started. Um and uh with [12:46] continuous delivery obviously you you [12:47] build your software, you test your [12:49] software and you deploy your software [12:51] and you deploy it you know as one big [12:53] deployment and that deployment could be [12:55] an update to your application and it has [12:57] a number of features underneath right so [12:59] you've got all your features baked into [13:00] that deployment you throw it over the [13:02] fence and then when it comes time to do [13:05] uh deployment testing you know maybe you [13:07] do a canary deployment where you can [13:09] actually deploy it to a small subset of [13:11] your users you do your canary [13:13] you see how things are going and then at [13:15] some point when things look good you can [13:18] uh basically uh deploy to everybody [13:20] right so that's kind of the typical way [13:22] things work with CD [13:24] however if something goes wrong with a [13:26] CD deployment what happens is you have [13:29] to do a roll back now harness has tools [13:31] to make automatic roll backs very easy [13:33] and and and painless at least compared [13:36] to doing them manually but it still [13:37] means rolling back all those features [13:39] that you just put out so whoever's been [13:40] exposed to those features that tend 20% [13:43] or something is going to notice maybe a [13:45] little bit of a difference um across [13:47] even the features that weren't affected [13:49] by wherever this problem took place. [13:51] This problem may may have taken place [13:52] downstream in a in a small feature [13:55] somewhere that was causing a security [13:56] vulnerability, but you had to roll back [13:58] everything, right? And uh take those [14:01] take those features kind of away for a [14:03] little while until you can fix things [14:04] and roll them out again. Um it's great [14:07] to have automated tools for that, but it [14:08] is kind of can be kind of painful to go [14:10] through that process. Um, with feature [14:13] flags, we can do things a little bit [14:15] differently. So, we can still do our [14:17] deployment, our big deployment, uh, but [14:19] if you bake feature flags in, then you [14:21] can treat each one of these features as [14:23] an if else statement in your code. And [14:26] if the feature flag is turned on, then [14:29] you can you can uh roll this out to a [14:31] very small subset. Most people start at [14:33] just 1%. I know I say 10 and 20 to make [14:35] things nice round numbers here, but [14:37] typically people will roll this out to [14:38] just 1% of your of your segment that [14:41] you're targeting. Maybe it's a [14:42] particular region, maybe it's power [14:44] users, maybe uh uh uh who knows, right, [14:48] what your segment are region or or or [14:50] something like that or maybe a platform [14:52] that you're rolling this out to. And you [14:54] can do this one at a time. And so what [14:56] happens is if you have a similar problem [14:58] where you've rolled out a small feature, [15:02] one of the small features in your [15:03] deployment to your power users or your [15:05] premium users and they start having a [15:06] problem, you can turn that flag off [15:09] without having to do a roll back. You [15:10] can just turn off that individual flag. [15:12] The rest of your deployment is [15:13] unaffected. Um, and you've essentially [15:17] rolled back instantly to the previous [15:19] state of your code because of the way [15:21] this works. So uh, feature flags can do [15:24] all that. Now what feature flags can't [15:25] do is they can't redeploy your entire [15:27] application. Um they can't roll back [15:29] that that entire update out all at one [15:32] time for you. But they can mitigate the [15:34] risk of a small subset of that [15:36] deployment causing a problem where you [15:38] would have to roll everything back. So [15:40] the best of both worlds here are to use [15:42] both, right? You use CD to go ahead and [15:44] do those deployments and then you use [15:46] feature flags to one at a time uh uh [15:50] deal with each one of these features. [15:52] Now why is this important? Why am I [15:53] talking about all this? Well, it's [15:55] because of that first problem I talked [15:57] about before where you have, you know, [15:59] 20 times the amount of features that you [16:01] have to release at any one given time. [16:03] So, we're trying to solve that problem [16:05] of things going wrong in that huge [16:08] amount of features, right, that we have [16:09] now. Not just one, not two, but maybe 20 [16:12] things, maybe a hundred things that we [16:13] rolled out at one go. And we need to be [16:17] able to have a way to deal with the [16:18] increased amount of exposure that we [16:20] have so that when something goes wrong, [16:22] we don't have to roll all of that back. [16:24] We can just roll one at a time. And that [16:26] certainly helps us mitigate that risk to [16:28] a certain extent. Um, so by combining [16:30] these two things, it's really the best [16:32] of both worlds. Now, let me give you a [16:33] more practical example. Maybe you have a [16:35] mobile app, right? [16:38] This is example of a banking app. My the [16:40] my bank. Uh, who who else banks at my [16:42] bank? I open all my accounts there. Um, [16:45] but you can imagine there's all this [16:47] competition out there, right? You can [16:48] see some of the logos over there on the [16:50] right. Everyone is vying for eyeballs. [16:53] Everyone is trying to come up with a [16:55] feature that will get uh their customers [16:57] or their potential customers to open [16:59] more bank accounts or use their app [17:01] more. And that's another great example [17:04] of how you can use feature flags and [17:06] deployments together as as one combined [17:09] way to deal with this AI code velocity [17:12] problem. As I as I mentioned in in in [17:15] the process of doing one single [17:17] deployment, you may have 180 features [17:19] kind of tucked behind there that you're [17:21] turning on one at a time. You're [17:23] probably experimenting with a subset of [17:24] those. Although again, I would argue you [17:26] should be experimenting with almost all [17:27] of them uh to know what's actually [17:29] impacting the bottom line. And then when [17:32] you uh once those features are [17:34] validated, you've run your experiments, [17:36] you know that they work, then the [17:37] banking app the next time it gets a [17:39] deployment to the next version, all [17:41] those things are codified in the code [17:43] and they become standard features, not [17:45] behind a feature flag anymore. Um, and [17:48] that's the way that a lot of people use [17:49] these tools. [17:54] Okay, so let's now talk about pipelines [17:57] real quick. Pipelines is uh people have [17:59] deployment pipelines. It's kind of a [18:01] generic term, but at harness it means [18:02] something a little bit more. We have our [18:04] own pipelines feature of the product. [18:07] The nice thing about it is it spans uh [18:11] uh multiple different products that we [18:14] sell, right? It's not just one tool for [18:16] one product. It actually integrates, for [18:19] example, with feature flags. I'll show [18:20] you an example of that in a minute. What [18:22] this allows you to do is it allows you [18:24] to very easily, you can set this up with [18:26] YAML if you want. uh you can very easily [18:28] deploy pipelines and uh do things in an [18:33] automated fashion with guardrails and [18:35] I'll give you some examples of that in a [18:37] minute. So you can see here we're doing [18:38] a verifi manual verification check which [18:41] uh looks like it failed in this case in [18:43] the graphic right so we're doing some [18:44] sort of verification maybe it maybe [18:46] there was a security problem or [18:47] something like that some sort of [18:48] verification happened um and that was uh [18:51] to protect us against releasing code [18:54] that was generated by by AI but may have [18:56] a problem with it um and we don't want [18:58] to release that into production yet. So [19:00] it's a great time to do these checks. [19:01] It's a great time to have these checks [19:03] in your pipeline. So we'll be talking a [19:05] lot about pipelines as we finish out the [19:07] presentation today. Okay. So we talked [19:10] about guardrails and validation. I'm [19:11] going to talk about those separately. So [19:12] what are the guardrails? What do [19:13] guardrails even mean? What are we [19:15] talking about? Well, my definition for [19:17] guardrails are the controls that decide [19:21] whether AI generated code and AI [19:23] assisted changes [19:25] are allowed to advance. So you you're [19:28] going to be making these things, but the [19:29] question is at what point do we do a [19:32] hard check to know if things are going [19:35] right or if things are going wrong? [19:36] Because nobody wants to end up in the [19:38] news for a security vulnerability that [19:40] leaked out, right? Okay. Well, what are [19:42] three ways we can deal with that? What [19:44] are three three types of guardrails we [19:45] can put up? Well, one, we can put up [19:47] human judgment. That would be kind of a [19:48] human in the loop problem where we at [19:50] least we might not need humans [19:52] evaluating every line of code whether [19:54] it's correct or not. But at least we [19:56] need some human checking to make sure [19:57] that this pull request for example meets [20:00] our standards or isn't going to break [20:01] something [20:03] before it goes into production. Also, we [20:05] need some sort of security evidence. So, [20:07] we need to do some sort of test. And [20:09] this also gets down to a little bit of [20:11] regulatory environments as well, right? [20:13] You need an audit trail, but we'll talk [20:15] about application security mostly here. [20:17] You need some sort of uh security check [20:19] to make sure that you don't have a [20:20] vulnerability. And then you need policy [20:22] enforcement because AI is really good [20:24] about coming up with its own workarounds [20:26] for for things. You know this if you've [20:29] used it and it'll say, "Well, I I just [20:31] created my own naming scheme for this [20:33] thing, right?" Well, no. If you're [20:35] working at a company with a thousand [20:36] people, you need a naming scheme that's [20:39] consistent across things so that the [20:40] incident team knows what's going on if [20:42] that there's a problem or or whatever. [20:44] So, you need some sort of way to enforce [20:46] policies. So, how do we do that? Well, [20:48] I'm going to tell you how we do it at [20:49] harness. You can um think about your own [20:51] products that you use when you're [20:53] thinking about this, but let's talk [20:54] about policy as code first and then [20:56] we'll work backwards. [20:59] So, as I said, uh you need some sort of [21:01] policy enforcement. The way that we do [21:02] it at harness is with OPA style policy. [21:06] So, we actually have a a policy um uh [21:10] when you go in to create a policy, we [21:12] have examples that you can use and we [21:14] show you the the code. I always forget [21:16] what the name of the OPA [21:19] standard is for writing policies, but we [21:21] use that same standard to make it easy [21:22] for people. So, you don't have to worry [21:24] about if you want to take these policies [21:25] to somewhere else later, they'll still [21:27] be valid. Um, and you can actually test [21:29] them. You can go in here and if you set [21:31] up a policy, let's use the same example [21:33] where I have a naming convention that [21:34] needs to be standardized on and I'm [21:37] setting up that policy. You can actually [21:38] go in and test put in a bad policy and a [21:41] good policy and see if the policy [21:42] enforcement works. [21:46] And here's an example of that [21:47] enforcement. So, we're running our [21:49] pipeline. Uh we're running the create [21:51] feature flag step in our pipeline, which [21:53] happens automatically as the pipeline [21:55] progresses. And we get to this point and [21:56] it says, "Hey, you've got a bad name in [21:59] here. Um, you've used the name Ola, [22:02] which doesn't require a Jira ticket [22:04] number." So, in this case, the company [22:06] wanted to have the Jira ticket ticket [22:08] number when as part of the feature flag [22:10] name. Again, I think this is probably [22:12] for, let's say, this goes to an incident [22:15] team. They need to be able to turn this [22:16] off because something happened. You want [22:19] to have that Jira ticket uh available to [22:21] you. And so, why not just put it right [22:22] in the name of the flag? Makes a lot of [22:24] sense. and you can enforce that across [22:26] your entire organization with policies. [22:29] So that's the first one. The second one [22:30] is human in the loop. How do we do that [22:32] or when do we use it, right? Um well, [22:35] you want to use it for approvals for [22:38] production. And some people are going [22:40] right to production to run certain [22:41] tests, right? Especially with their user [22:43] interface, uh with their user [22:45] experiment, uh experience, they want to [22:47] do that in production. But before [22:48] something goes to production, you [22:50] certainly want to have at least somebody [22:52] that is familiar with the code to go [22:53] through and make sure that it's not [22:54] going to break something else, right? [22:56] What about high-risk services, financial [22:58] transactions or uh regulated data um [23:02] that uh uh needs to be handled [23:04] differently? Um unusual deployment [23:07] patterns. So you you know, you've all [23:08] been through this where you're starting [23:10] up a new service maybe in the cloud or [23:11] something, you haven't done it before. [23:13] you definitely want to have somebody [23:15] validating that to make sure that things [23:17] are right there as well. And then of [23:19] course, if you get to some sort of [23:20] failed verification that I showed you [23:22] before, right? We're trying to validate [23:24] u a featured, run a security check, [23:26] something fails, you want to be able to [23:28] have a human go in and investigate [23:30] what's going on. [23:32] Uh so what do you use for that? How do [23:34] you get notified that something's wrong? [23:35] Well, if you're using a product like [23:37] Harness that we've built in tools to [23:39] basically integrate with whatever you're [23:40] using. So, you can obviously be [23:42] notified. This is an example of an inapp [23:44] notification where someone needs to go [23:46] into harness and put in yes, I validated [23:48] that each feature is working, click [23:51] approve, everything looks good, and then [23:53] the uh the pipeline will automatically [23:55] continue if they hit approve at this [23:56] point. Uh but also you need to integrate [24:00] with your existing ticketing tools or [24:02] your incident systems that you have your [24:04] service nows etc your pager duties and [24:06] messaging tools like Slack. Those are [24:08] all ways that people use uh to make sure [24:11] they're notified so that not only is [24:13] there a stop an automated stop in your [24:16] pipeline to say I need to go validate [24:18] this particular part uh but you're [24:20] notified that that you need to go do [24:22] that in whatever system works for you. [24:24] So uh really important to keep those in [24:26] place [24:29] and we can't forget security testing. So [24:31] we have application security testing [24:33] tools um that can go out and again be [24:36] integrated into these pipelines along [24:38] with everything else. Uh really [24:40] important here is dduping [24:41] vulnerabilities. So one thing that we're [24:43] seeing with AI code is if there is a [24:45] vulnerability and you commit that to a [24:47] centralized repo as part of a [24:50] instruction set or something that's [24:51] going to reproduce that vulnerability [24:53] over and over again those [24:54] vulnerabilities can get propagated. So [24:56] by having a security testing tool like [24:58] we deliver at harness you can actually [25:00] get in front of that and and uh prevent [25:03] those uh duplication of vulnerabilities [25:05] uh by AI which is super easy to do. Um [25:10] and then we also have remediation. So, [25:12] you know, enforcing that governance with [25:14] those same OPA policies we talked about [25:15] before, being able to track issues by [25:18] integrating into Jira and Slack, for [25:19] example. Same sort of notification stuff [25:22] I was talking about before applies here. [25:23] So, you get notified immediately. You [25:26] can go in and say, "Hey, uh, you can ask [25:28] our AI system, for example, I, you know, [25:30] I saw that there was a problem. What's [25:32] going on?" It can come back and and [25:33] pinpoint the problem for you so that you [25:35] can uh, uh, remediate it quickly. Super [25:38] important to have this as a guardrail [25:40] when you're rolling out AI generated [25:41] code. [25:44] Now, let's talk a little bit about [25:45] validation. Again, what do I mean by [25:47] validation? Well, what I'm talking about [25:49] here is the process of proving that a [25:52] release is safe, useful, worth keeping, [25:56] it's important, and codifying. Again, if [25:59] you have a better word for that, let me [26:01] know. I don't know what it is. [26:02] Validating, codifying. Basically, you [26:03] need to go in and and and um fix your [26:06] code or or uh codify your code to the [26:09] point where you've taken the feature [26:11] flag out. The process that you validated [26:13] is working and is important for your [26:14] business is now in the next release, [26:17] right? It's in the next deployment. So, [26:19] what are we talking about there? What [26:20] kinds of things fall under validation? [26:22] Well, for me, it's validating the [26:23] release process itself. It's integrating [26:25] this into the release process so that [26:27] you can keep things moving quickly and [26:29] you don't get bottlenecked with all of [26:31] this AIdriven code that's coming your [26:32] way. Experimentation again got to make [26:35] sure that this stuff is impacting the [26:37] bottom line or affecting your users in [26:38] the way that you want. And then [26:40] obviously tech debt is a big deal. Tech [26:42] debt can increase uh with right along [26:46] with the number of changes that you're [26:47] putting out uh due to AIdriven code [26:49] velocity that tech debt is also [26:52] accelerating uh because of that as well. [26:54] So if you're not taking steps to clean [26:55] it up, you're going to run into [26:56] problems. [27:00] So let's first start about how we can [27:02] deal with that release process and [27:04] feature flags. I gave you all the [27:05] examples of this. I'm not going to go [27:06] into too much more detail here, but [27:09] basically you can think of it as uh um [27:12] you know the last bullet here, limiting [27:14] the blast radius, right? So, one of the [27:17] things that you want to do as you're [27:18] releasing more and more features is to [27:22] make sure that the impact, whatever [27:23] impact you have, if there is a problem, [27:25] that it's limited. It's limited to maybe [27:27] your internal users only. Wouldn't that [27:29] be great? You release this to internal [27:31] users. Have them take a look at it [27:32] first. It's not going to catch [27:33] everything, but it's certainly not going [27:35] to land you in the newspaper or on uh uh [27:39] your your your daily Google newsfeed, [27:42] right? So, not only that, but you can [27:44] roll out progressively by user, by [27:46] segment, by region. As I mentioned [27:47] before, doing a progressive roll out is [27:50] much safer um and can be much more [27:52] effective than rolling out to everybody [27:54] all at once. And then, of course, [27:55] instant kill switches. That's a great [27:57] way to look at feature flags. This whole [27:59] area of feature flags has developed. [28:01] used to be really for, you know, people [28:03] that were wanted to open up a beta, for [28:05] example, to some uh customers and they [28:08] would turn on the feature flag and the [28:09] and the the customer in in question [28:12] would get the the beta of a new feature [28:14] or new new uh uh product or something [28:17] like that, right? It's evolved u much [28:20] more since then and now it's used to do [28:23] all sorts of things and especially to [28:24] guard against problems when you're doing [28:26] these AI a lot more AI generated [28:29] releases like this [28:33] talk about experimentation as well is [28:35] there a bottleneck with experimentation [28:37] well what we found is that for advanced [28:40] experimentation teams uh even if they've [28:43] done everything right so they've built [28:46] up their experimentation team they've [28:48] created dashboards, they're doing [28:49] dynamic configs, they're uh plugged into [28:52] warehouse native, they've shared this [28:54] learning experience across their team, [28:56] they still hit a bottleneck. And the [28:58] problem is exactly what we've been [29:00] talking about. They build it to a point [29:02] where maybe they're doing experiments [29:03] weekly, but they can't get beyond that [29:05] because they just can't run any more [29:07] experiments any faster. And what we [29:09] found the bottleneck is here is actually [29:12] the uh same thing that we've been [29:14] talking about which is they aren't [29:18] building the experimentation platform [29:19] into the rest of their deployment [29:22] pipeline. So, uh, if you're using a [29:25] separate experimentation tool, that's [29:27] fine. Just know that you may run into [29:29] this bottleneck problem where you hit a [29:32] certain ceiling and you're not able to [29:35] keep up with the code velocity that's [29:37] that's coming your way, especially if [29:38] you're changing your [29:41] expectation with experimentation to be [29:44] experimenting on everything, right? [29:45] Think about your experimentation teams [29:47] here, right? they they they were maybe [29:49] running, you know, 10 experiments or [29:50] something like that, you know, now [29:51] they're running a hundred experiments. [29:53] How are they going to keep up with that? [29:55] How are they going to automate that? Um, [29:57] it can be a real problem. And and the [29:59] best answer that we found is to make [30:01] sure that the tools are integrated. [30:04] Uh, the experimentation tool, your [30:06] feature flagging tool, your CI/CD tool, [30:09] make sure all of those are integrated. [30:10] And that will take out the bulk of the [30:12] of the bottleneck because you can [30:13] automate all the way through that whole [30:15] pipeline. [30:18] Uh, now let's talk about removing tech [30:20] debt. So, again, I'm going to be [30:21] focusing on feature flags here because [30:23] that's my bread and butter. [30:28] The point is you need to be doing this [30:30] uh because it's going to come back and [30:31] haunt you. Now, what we've done for [30:33] feature flags is we've actually released [30:35] an an automated uh feature flag cleanup [30:38] tool that will go out and it will based [30:41] on your guidelines, it will go out and [30:44] automatically find feature flags that [30:46] are let's say in this case 100% rolled [30:49] out, right? So, you don't want to be [30:50] removing feature flags that haven't been [30:52] fully released yet. So, as you're [30:54] ramping up, you don't want to be release [30:56] getting rid of that feature flag code. [30:57] But what you can do is once it's been uh [31:00] fully rolled out, the experiment is [31:01] finished, you've proven the value of [31:03] that particular feature, you can have [31:06] this tool go ahead and run and it will [31:08] actually go in and identify in your code [31:10] where those feature flags are and set up [31:13] uh automated ways to actually uh uh pull [31:16] that code out and including, you know, [31:19] doing a uh uh uh a PR for you on your [31:22] behalf and things like that. We're [31:24] always improving this feature. It's it's [31:27] uh hot off the press, if you will. So, [31:29] we're getting a lot of feedback on it [31:30] and a lot of good feedback. And uh this [31:33] is something that we feel can really [31:34] help people go in and clean up their [31:36] code because the last thing you want to [31:37] have is old code hanging out in your co [31:39] codebase that an AI tool can reference [31:42] and perhaps make assumptions on your [31:44] behalf that you don't want it making. [31:46] So, cleaning up this code is really [31:47] important. [31:50] Okay, let me give you an example of a [31:51] company that's seen dramatic results by [31:54] uh following the approach that I just [31:56] laid out, right? Especially with the [31:58] integrated pipelines, being able to [32:00] automate things, having enough review in [32:03] the loop, uh human review and and [32:06] guardrails and things like that that [32:07] they can move very very quickly. So, [32:08] this is Speedway Motors. We've actually [32:11] uh done another lead dev webinar with [32:13] them. I'll give you a link to that in [32:14] just a minute. Um, but before they [32:16] started using our tools, they were only [32:19] releasing a couple times a month. That's [32:21] as fast as they could do it. And because [32:24] they weren't didn't really have the uh [32:26] the right tools in place to run [32:27] experiments to know what was impacting [32:29] their e-commerce tools or the e-commerce [32:31] uh product that they have, they weren't [32:33] really able to see any increase in [32:35] conversion rates when they did do [32:37] testing or when they did release new [32:39] features. So once they started using our [32:42] tools, they found very quickly that they [32:44] got immediate ROI because they were able [32:46] to go from releasing two times a month [32:48] to releasing multiple times per day um [32:51] sometimes two, three times per day. So [32:53] their deployment frequency went way up. [32:56] They were also able to see an immediate [32:58] impact and they were able to make [32:59] judgments on whether new features were [33:01] helping or hurting and then move forward [33:03] very quickly and get those uh changes in [33:06] their codebase and get them rolled out [33:07] uh in the next deployment which was [33:09] really important. Um and they found that [33:11] there was a big impact to the bottom [33:13] line uh because they had the visibility [33:15] that they needed because they didn't [33:17] have separate tools getting in the way [33:19] of each other and a lot of delay. they [33:21] were actually able to iterate quickly [33:22] and say, "Hey, this particular special [33:25] that we ran on the website or the way [33:26] that we did this is actually causing [33:28] people to order more stuff and so we [33:30] need to do that again and again, you [33:31] know, every year seasonally, whatever it [33:33] is, whatever the problem is." But they [33:34] were able to to validate that that was [33:36] working, which is really great. [33:39] Okay, let me go ahead and give you some [33:41] resources now, things that you can take [33:43] with you. Uh, these will be offered to [33:44] you as well after the webinar, but I [33:47] want to talk about a few things. So, [33:48] coming up on June 17th is uh our annual [33:52] flagship event that we do at Harness. Uh [33:55] Split used to do this before we acquired [33:57] them and we've kept it going and it's [34:00] special because this isn't just us [34:02] talking to you about what we think. What [34:04] we do is we actually go out and get [34:06] actual practitioners and have them come [34:07] in, practitioners that are doing [34:09] incredible things and they talk about [34:11] the problems that they're facing and [34:12] they talk about how they're solving [34:14] them. Uh whether it's with harness or [34:16] not, it doesn't matter. We have some [34:17] folks coming in from Curser to talk [34:19] about the AI revolution and what that [34:21] means. We've got uh some folks coming in [34:23] from some financial institutions, some [34:25] retail companies. They're coming in to [34:27] talk about how they're dealing with this [34:29] AI velocity problem, what they've put in [34:32] place, how they're running experiments, [34:33] how they're cleaning up their code. Um [34:35] all of that is going to get talked about [34:37] by practitioners. So, uh you can go out [34:40] and sign up for this. Just do a Google [34:41] search for Harness Flagship 2026 should [34:44] come up and you can register for that [34:46] event. And be sure to register even if [34:48] you can't attend the whole thing. It's [34:50] free. It's virtual. Um you can listen to [34:53] it in the background, but if you get [34:54] pulled away or whatever, you'll have the [34:56] replays available to come back and [34:58] listen to as well. I don't think you're [34:59] going to want to miss it. And uh [35:01] definitely go out and register for that [35:03] event. [35:05] Um I also mentioned this report that we [35:07] put out along with lead dev. So, [35:09] there'll be a link to download that [35:11] report. I I recommend going out and [35:13] doing that and kind of seeing where you [35:14] fit in. I can tell by your poll [35:16] responses that uh you're going to fit [35:18] right in with the responses on this [35:20] survey. But there are some interesting [35:22] takeaways. Um, and we also talked about [35:24] this in a little bit more detail on a [35:26] webinar called d-risking releases in the [35:28] AI era. And so, I think I was on that [35:31] one. And, uh, we talked about what the [35:33] implications are. We kind of drilled [35:35] into some of the details, some of the [35:36] findings from that report in a webinar [35:39] fashion. So again, great listening [35:41] material. If you're working or doing [35:42] something around the house and you want [35:44] to hear the the output of that report [35:46] and what the implications are, highly [35:48] recommend that one. And I talked about [35:49] Speedway Motors. We also have uh that [35:52] webinar. You can go back and listen to [35:54] them. Again, practitioners, these are [35:55] the people from Speedway Motors that [35:56] implemented these things. Definitely go [35:59] out and take a look if you're [36:00] interested. One of the things they [36:01] talked about which I did not realize [36:03] until we were on that webinar talking [36:04] about it was um they even control some [36:07] of their physical uh resources through [36:11] feature flags. So I think the example [36:13] was they control their warehouses. If [36:15] there's inclement weather and the [36:18] warehouses shouldn't be running, they [36:19] actually flip a feature flag to turn off [36:21] the warehouse, which is uh which is [36:24] amazing to me. but it's an easy way for [36:26] them to control some of those things uh [36:28] phys in the physical world as well as in [36:30] their application. [36:34] Um, and last but not least, uh, if you [36:36] want a demo of any of these things, if [36:38] any of these topics that we talked about [36:39] through guardrails and validation, if [36:42] any of those are problems for you right [36:43] now and you just want a a a demo of how [36:46] it would work, that's more specific to [36:49] your particular use case or the problem [36:51] that you're facing, uh, you can scan [36:53] this QR code or just, uh, go out to the [36:56] website and sign up for a demo. Highly [36:57] recommend you do that. There's no cost, [36:59] no risk. We'll just ask you a few [37:01] questions. will get on the phone and [37:02] we'll walk through the product in a way [37:04] that's more applicable to you. So, go [37:06] sign up for that demo and uh I'd love to [37:09] see some of the folks on this call [37:11] actually uh in the future perhaps on a [37:13] demo call. That would be great. All [37:15] right, [37:17] so for now, uh let's go ahead and get [37:20] over to [37:22] some questions. [37:26] See if I can bring up my Q&A here. [37:29] Uh this is the time if you do have any [37:32] questions [37:34] uh go ahead and put them in now. We'll [37:35] take a few more minutes. I do have one. [37:37] Uh do you support rolling back database [37:39] changes? Um yes we do actually. We can [37:42] help with uh uh we actually have a tool [37:44] called um DB DevOps and the DB DB DevOps [37:49] tool. I have to be careful because I [37:50] want to say DevOps. The DB DevOps tool [37:53] actually can help you rolling back your [37:55] DB changes or doing schema migrations. [37:58] um, you know, it's not my particular um, [38:03] uh, product that I work on, but I do [38:06] know that we do have it. It does [38:07] integrate with the pipeline process that [38:09] I showed you before. So, as part of your [38:11] process, you're rolling out a new [38:12] feature, you need to move someone over [38:14] or do some sort of a database uh, [38:16] change, you can actually build that into [38:18] the pipeline and have it happen [38:20] automatically. [38:26] Um, [38:27] I'll give it just one more second to see [38:29] if there's more questions that came come [38:31] in. [38:33] One of the interesting ones that came in [38:35] uh last time when I was talking about [38:36] this was how do I convince my management [38:38] that this part of the process is [38:40] important. [38:42] Um, that that can be a struggle right [38:43] now. I think a lot of people are [38:45] focusing on the code development side of [38:48] things and how many tokens you're using [38:50] and how much code you're pushing out. [38:52] Um, but they're starting to realize that [38:54] that code doesn't always make it all the [38:56] way out into uh production because of [38:59] this deployment problem that we've been [39:01] talking about and the release and and [39:03] experimentation problem that happens [39:05] down the line. Um, it's not just that [39:07] you're developing more code, it's that [39:09] you're doing more changes and you need [39:11] to be running experiments on more stuff. [39:13] So, if you don't have a good tool, uh [39:16] especially if you're running at a large [39:17] organization with thousands of [39:19] developers, if you don't have a good [39:20] tool to help you with the rest of the [39:22] process, um you could be in for a world [39:25] of of hurt. And uh you might not get the [39:28] results that the executives want to see. [39:30] You might get the results that your [39:31] manager wants to see, but the executives [39:32] might not be too pleased that there's [39:34] all these changes backed up in the [39:35] pipeline. [39:37] Um [39:40] let's just see if there's anything else. [39:44] Uh let's see. How do you put in [39:46] guardrails for uh safe handling PII or [39:50] other security guardrails while pushing [39:52] changes? This is a good good question. I [39:54] talked about application security and we [39:57] talked a little bit about regul [39:58] regulated environments. Um you can [40:00] there's a couple of things I'll mention [40:02] here. So you can definitely put in [40:04] guardrails as part of policies for [40:06] dealing with PII. That's probably the [40:08] the first way that I would handle that [40:10] is put in a policy that says, you know, [40:13] this this shouldn't this database [40:14] shouldn't be accessed or this uh type of [40:17] information shouldn't be shared. This uh [40:20] credit card uh sequence of numbers [40:22] shouldn't be should never be used or [40:24] there's policies that we have in place [40:25] for for all of those things. Um and of [40:28] course, every company's going to be a [40:29] little different. So, they're flexible [40:30] enough you can create your own policy to [40:32] make sure that that's not happening. Um [40:34] flexibility is key. That's why we offer, [40:37] you know, YAML, you can do pipelines as [40:40] code, for example. Um, [40:43] and and so, yeah, it's very flexible. [40:45] Putting in those policies for that [40:47] information that shouldn't be shared is [40:48] not difficult, and it definitely should [40:50] be part of a check as some at some point [40:52] and as part of your uh pipeline for [40:55] sure. [40:59] Um, [41:02] just looking at the questions here, [41:06] one more. How do you navigate a testing [41:08] team who wants to test every permutation [41:12] of a feature flag? Basically, if we have [41:15] multiple feature flags in flight, we [41:17] need to ensure tuning them uh turning [41:19] them off is going to work just because [41:21] it's coded behind if else there are [41:24] still things that can go wrong. Uh, I'm [41:27] going to have to interpret this one a [41:28] little bit. Um, [41:32] if you you you need a tool that supports [41:34] being able to test every permutation. [41:36] Um, it's so obviously there's going to [41:38] have to be a conversation about how much [41:41] testing can take place. Um, I'm assuming [41:43] this is a velocity question, right? [41:45] Like, oh, we haven't tested uh this [41:47] particular use case or this particular [41:50] thing, so we can't move forward yet. Um, [41:52] you can run multiple tests at once. You [41:54] can have multiple feature flags turned [41:56] on. You can have multiple segments [41:57] turned on. You can have multiple [41:59] treatments turned on. And you can be [42:00] doing multiple experimentation turned [42:02] on. There was a demo I did the other day [42:04] where we showed all the different [42:05] permutations that we had running at the [42:07] same time. And I think it was 256. I [42:09] don't know. I don't know if there's an [42:10] upper limit, but there was 256 different [42:12] things being tested. And then [42:14] automatically we were able to decide [42:16] which one of those was working and which [42:18] ones weren't. And then we were able to [42:20] progress that roll out very quickly and [42:22] determine what the best one that [42:24] basically won the evaluation contest um [42:28] so that we could roll that out more [42:29] broadly. So I know that we can do it at [42:30] harness. Um hopefully your [42:32] experimentation tool is able to do that [42:34] as well and hopefully it's integrated [42:35] with the rest of your pipeline to make [42:37] that process easy. Um okay, I think [42:39] we're about out of time and I think [42:41] that's all the questions we have for [42:42] today. So thanks everybody for joining. [42:44] Thanks for spending a little bit of time [42:46] with me. Really appreciate it. I hope I [42:48] get to talk to you on one of these in [42:49] the future and I hope you have a great [42:51] day. Thanks a lot and we'll see you [42:53] later. Bye-bye.  ## 📺 Video ![](https://www.youtube.com/watch?v=Xt7y6CKDqcI) ## Highlights - We're seeing four to 20 times the number of changes being pushed out the door or at least through the process that we were seeing before. And we all know what happens when we have an increased number of things changing at any given time. That opens up the risk that something's going to go wrong. - Note: 📚 AI code velocity creates deployment bottlenecks without proper guardrails · - Core problem statement around 8:30 in presentation - Why this matters: Identifies the fundamental mismatch between AI coding speed and deployment capability · #atom/literature/h1 - If you have a similar problem where you've rolled out a small feature, one of the small features in your deployment to your power users or your premium users and they start having a problem, you can turn that flag off without having to do a roll back. You can just turn off that individual flag. - Note: 📚 Feature flags enable granular rollback without affecting entire deployments · - Technical explanation around 15:45 showing CD vs feature flag approaches - Why this matters: Solves the blast radius problem when AI generates many features per deployment · #atom/literature/h2 - What are three types of guardrails we can put up? Well, one, we can put up human judgment. Also, we need some sort of security evidence. And then you need policy enforcement because AI is really good about coming up with its own workarounds for things. - Note: 📚 Three guardrail types required for AI code validation · - Framework presentation around 22:10 defining comprehensive guardrail strategy - Why this matters: Provides actionable taxonomy for implementing AI code safety measures · #atom/literature/h3 - AI-generated feature velocity requires proportional deployment sophistication - Note: [[DevOps transformation patterns]] — because extends existing CD practices · - [[DevOps transformation patterns]] — because extends existing CD practices · #atom/zettel/z1 - Progressive rollout percentages (1% → segments → full) need optimization research for different feature types - Note: ✏️ Fleeting capture #atom/fleeting/f1 - Integration patterns between WordPress deployment tools and feature flag platforms worth exploring - Note: ✏️ Fleeting capture #atom/fleeting/f2 - Policy-as-code examples for AI coding standards could be valuable community resource - Note: ✏️ Fleeting capture #atom/fleeting/f3 --- # 📝 Full Notes (Deep-Read Analysis) # 🏷️ Title: How to build the guardrails your AI releases actually need ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ## 📝 Summary - **What happened**: AI coding tools are increasing feature velocity 4-20x, but organizations lack proper guardrails for testing and deployment phases. - **Why it matters**: Without proper controls, AI-generated code creates security vulnerabilities, quality issues, and deployment bottlenecks despite faster development. - **Key stakeholders**: Development teams, DevOps engineers, security teams, product managers, experimentation teams. - **Risks and tradeoffs**: Faster coding vs. increased deployment risk; automation efficiency vs. human oversight needs; feature velocity vs. technical debt accumulation. - **Bottom line**: Success requires combining continuous delivery with feature flags, implementing three types of guardrails (human judgment, security evidence, policy enforcement), and validating releases through experimentation. ## 📖 Main Content The presentation addresses a critical gap in AI-assisted software development: while organizations excel at providing AI coding tools, they struggle with the downstream effects of dramatically increased feature velocity. Eric Nukem from Harness identifies that teams are seeing 4-20x more features in deployments but lack adequate systems for safe release management. ### The AI Code Velocity Problem Current AI coding adoption shows strong uptake in development phases—developers use Claude, Copilot, Cursor, and ChatGPT effectively. However, survey data reveals concerning gaps: - Most organizations still require human review of every line of AI-generated code - Less than half have implemented AI guardrails - Only 36% report AI improving software quality - 58% are running more experiments but systems can't handle the volume ### Solution Framework: Guardrails + Validation The presentation proposes a two-pronged approach focusing on testing/deployment and release/experimentation phases: **Three Types of Guardrails:** 1. **Human judgment**: Approvals for production, high-risk services, unusual deployment patterns, failed verification scenarios 2. **Security evidence**: Application security testing with vulnerability deduplication and remediation workflows 3. **Policy enforcement**: OPA-style policies ensuring naming conventions, compliance standards, and organizational consistency **Validation Process:** - Release process integration with feature flags - Progressive rollout capabilities (1% → segments → full deployment) - Experimentation on every feature release - Technical debt cleanup through automated feature flag removal ### Continuous Delivery + Feature Flags Integration The core technical approach combines traditional CD deployment with granular feature flag control. While CD handles application-level deployments, feature flags enable individual feature rollouts with instant kill switches. This reduces blast radius from full rollbacks affecting all features to targeted flag toggles affecting only problematic features. ## 🗒️ Notes #### ⏱️ [00:03:12](https://www.youtube.com/watch?v=Xt7y6CKDqcI?t=192) AI Development Workflow Problems - Coding and review phases well-supported with AI tools - Deployment size explosion: 4-20x more features per release - Testing and deployment become bottlenecks - Release and experimentation validation critical for AI-generated features #### ⏱️ [00:08:30](https://www.youtube.com/watch?v=Xt7y6CKDqcI?t=510) Survey Results on AI Adoption - Human review still required for every line of AI code - <50% have AI guardrails implemented - 58% running more experiments but systems can't scale - Only 36% see quality improvements from AI #### ⏱️ [00:15:45](https://www.youtube.com/watch?v=Xt7y6CKDqcI?t=945) Continuous Delivery vs Feature Flags - CD: Build → Test → Deploy as single unit with canary rollouts - Problem: Rollback affects all features in deployment - Feature flags: Individual feature control with progressive rollout - Best practice: Combine both for deployment flexibility and risk mitigation #### ⏱️ [00:22:10](https://www.youtube.com/watch?v=Xt7y6CKDqcI?t=1330) Three Guardrail Types - Human judgment: Production approvals, high-risk services, unusual patterns - Security evidence: Vulnerability scanning, deduplication, remediation - Policy enforcement: OPA policies for naming conventions and compliance #### ⏱️ [00:28:15](https://www.youtube.com/watch?v=Xt7y6CKDqcI?t=1695) Validation Framework - Release process integration prevents bottlenecks - Progressive rollout: Internal users → segments → full deployment - Experimentation requirement on all features - Technical debt cleanup through automated flag removal #### ⏱️ [00:35:20](https://www.youtube.com/watch?v=Xt7y6CKDqcI?t=2120) Speedway Motors Case Study - Before: 2 releases/month, no conversion rate visibility - After: Multiple daily releases, immediate ROI measurement - Integration enabled rapid iteration and business impact validation ## 📚 References - [Harness Flagship 2026 Event](https://www.harness.io/flagship) - [D-risking Releases in the AI Era Report](https://www.harness.io/resources) - [Speedway Motors Implementation Case Study](https://www.harness.io/customers/speedway-motors) - [Harness Product Demo](https://www.harness.io/demo) ## 🗣️ Paraphrase Organizations are experiencing a paradox with AI-assisted development: while coding velocity has increased dramatically, the benefits aren't reaching production due to inadequate release management systems. The solution isn't to slow down development but to implement proper guardrails and validation processes that can handle the increased throughput. The key insight is that traditional deployment approaches—treating all features as a monolithic release—become untenable when AI generates 20x more features. Feature flags provide the granular control needed to manage this complexity, allowing teams to validate individual features without risking entire deployments. However, this requires integrated tooling that spans the entire pipeline from development to production. ## 📰 Newsletter **AI Code Velocity: The Hidden Deployment Crisis** Your developers are shipping code faster than ever with AI assistance, but are those features actually reaching users? A new Harness survey reveals that while AI coding tools see widespread adoption, most organizations lack the guardrails needed for safe, rapid deployment. The problem: AI-generated features create 4-20x larger deployments, but traditional rollback strategies affect all features when problems arise. The solution combines continuous delivery with feature flags, enabling granular control and instant kill switches for individual features. Key takeaways for WordPress professionals: - Implement progressive rollout strategies (1% → segments → full) - Require experimentation on every feature release - Automate technical debt cleanup to prevent AI-generated cruft - Integrate security scanning into deployment pipelines Tools to explore: Feature flag services, experimentation platforms, policy-as-code frameworks, automated cleanup utilities. ## ✅ Action Items #### DevRel - Create educational content on feature flag implementation patterns for AI-generated code - Develop workshops on progressive rollout strategies and blast radius reduction - Build demo environments showing CD + feature flag integration - Document security scanning integration patterns for AI code validation #### OSPO - Establish policies for AI-generated code review requirements in contribution guidelines - Implement security scanning requirements for AI-assisted contributions - Create governance framework for feature flag naming conventions across projects - Define SLAs for AI code vulnerability remediation processes #### WordPress - **Core**: Evaluate feature flag integration patterns for core feature releases - **Core Performance**: Implement progressive rollout for performance optimizations - **Security**: Establish AI code scanning requirements for security-sensitive areas - **Community**: Create guidelines for plugin developers using AI coding tools - **Hosting**: Develop recommendations for hosts managing AI-generated plugin deployments #### The WPCC - Fund research into WordPress-specific feature flag implementations - Support development of AI guardrail templates for WordPress projects - Create grants for experimentation platform integration with WordPress workflows ## 🧪 Hypotheses 1. **Integration Hypothesis**: Organizations using integrated CD + feature flag platforms will achieve 2-5x faster time-to-production for AI-generated features compared to those using separate tools. 2. **Guardrail Effectiveness**: Teams implementing all three guardrail types (human judgment, security evidence, policy enforcement) will reduce AI-related production incidents by 70-90% within 6 months. 3. **Experimentation Scaling**: Feature flag integration enables experimentation teams to scale from weekly to daily experiment launches without increasing team size. 4. **Technical Debt Prevention**: Automated feature flag cleanup reduces AI-generated technical debt accumulation by 60% compared to manual cleanup processes. 5. **Progressive Rollout Impact**: 1% → segment → full rollout patterns reduce blast radius of AI code issues by 95% compared to full deployment strategies. ### 📊 Methodology - A/B testing comparing integrated vs. separate toolchain outcomes - Longitudinal studies tracking incident rates before/after guardrail implementation - Survey research on experimentation team capacity and throughput - Code quality metrics analysis for cleanup automation effectiveness - Case studies measuring deployment risk reduction through progressive rollout ### ❓ Questions - How do feature flag naming conventions scale across large organizations with thousands of developers? - What are the optimal progressive rollout percentages for different types of features (UI, API, infrastructure)? - How can teams balance automation with human oversight without creating approval bottlenecks? - What security scanning tools are most effective at catching AI-generated vulnerabilities? - How do you measure the ROI of guardrail implementation versus deployment velocity? ### 🔮 Future Research - AI-assisted feature flag management and automated rollout decisions - Integration patterns between WordPress-specific deployment tools and feature flag platforms - Security vulnerability patterns specific to AI-generated WordPress plugins and themes - Cost-benefit analysis frameworks for guardrail implementation - Machine learning approaches to predicting optimal rollout strategies ### 💡 Implications - **Practice**: Development teams need to expand focus beyond coding to include deployment pipeline optimization - **Policy**: Organizations require new governance frameworks specifically addressing AI-generated code risks - **Security**: Traditional security review processes may be inadequate for AI code volume and unique vulnerability patterns - **Education**: Developer training must include feature flag management and progressive rollout strategies - **Funding**: Investment in integrated toolchains provides better ROI than point solutions for AI development workflows ### 🧠 Learning Styles Assessment - **Reading level estimate**: Flesch-Kincaid Grade Level ~12 (technical business presentation) #### 👀 Visual learners - **Present**: Architecture diagrams showing CD + feature flag integration, pipeline flow charts, deployment percentage visualizations - **Improve**: Add before/after deployment risk diagrams, feature flag decision trees, security scanning workflow visuals #### 🔊 Aural learners - **Present**: Webinar format with presenter narration and Q&A segments - **Improve**: Provide audio-only version, add discussion group integration, create podcast-style follow-up content #### 🗣️ Verbal learners - **Present**: Clear definitions of technical terms, analogies (banking app example), structured explanations - **Improve**: Add glossary of terms, provide story-driven case studies, include plain-language summaries #### 🧩 Kinesthetic learners - **Present**: Demo offers, hands-on tool exploration suggestions, QR codes for immediate action - **Improve**: Create guided lab exercises, provide configuration templates, offer sandbox environments #### 🧠 Logical learners - **Present**: Structured framework (guardrails + validation), survey data, measurable outcomes - **Improve**: Add decision matrices, ROI calculations, step-by-step implementation checklists #### 🧑‍🤝‍🧑 Social learners - **Present**: Q&A interaction, community event promotion, peer examples (Speedway Motors) - **Improve**: Include discussion forums, peer review components, collaborative implementation guides #### 🧘‍♀️ Solitary learners - **Present**: Self-paced webinar format, downloadable resources, individual demo scheduling - **Improve**: Add self-assessment tools, provide reflection prompts, create independent study materials ### 📊 Accessibility Assessment #### Screen reader semantics - **Present**: Structured presentation with clear headings and sections - **Improve**: Add detailed alt text for diagrams, ensure slide content is screen reader accessible #### Keyboard navigation - **Present**: Standard webinar platform navigation - **Improve**: Verify all interactive elements are keyboard accessible, provide navigation shortcuts #### Contrast and readability - **Present**: Professional slide design with adequate contrast - **Improve**: Verify WCAG 2.2 AA compliance for all visual elements, test with high contrast modes #### Media and captions - **Present**: Live webinar with Q&A capability - **Improve**: Provide auto-generated captions, offer transcript downloads, ensure audio quality ### 🌍 DEIB Assessment #### Cultural breadth - **Present**: Focus on technical practices applicable across organizations - **Improve**: Include examples from diverse company sizes and industries, consider global deployment patterns #### Inclusive language - **Present**: Professional, technical language appropriate for business audience - **Improve**: Define technical jargon consistently, provide alternative explanations for complex concepts #### Neurodiversity support - **Present**: Structured presentation with clear sections and predictable flow - **Improve**: Provide content outline upfront, use consistent formatting, offer multiple content formats #### Representation - **Present**: Business-focused content with practical examples - **Improve**: Include diverse organizational examples, acknowledge varying team structures and capabilities ### ✅ Logical Validity #### Strong reasoning - **Deductive reasoning**: Clear logical progression from AI code velocity increase → deployment complexity → need for guardrails - **Inductive reasoning**: Survey data supporting conclusions about industry-wide challenges - **Causal reasoning**: Direct connection between toolchain integration and improved outcomes demonstrated through case study #### Fallacies - **Selection bias**: Speedway Motors case study may not be representative of all organizations - **Solution bias**: Presentation heavily favors Harness-specific toolchain without adequate comparison to alternatives - **False dichotomy**: Implies organizations must choose between speed and safety rather than exploring other approaches #### Tone of voice - **Professional and authoritative**: Uses industry terminology and data to establish credibility - **Solution-oriented**: Focuses on practical implementation rather than theoretical concepts - **Vendor-neutral framing with product integration**: Acknowledges multiple tools while showcasing Harness capabilities ### 🛡️ DARVO Analysis - **Deny**: No denial patterns present - speaker acknowledges industry-wide challenges directly - **Attack**: No attacks on critics or alternative approaches - maintains professional tone throughout - **Reverse victim/offender**: No victim/offender reversal detected - presents collaborative problem-solving approach ## 🎯 Conclusion AI-generated code velocity creates a new category of deployment risk that traditional release management approaches cannot adequately address. The solution requires integrated toolchains combining continuous delivery with feature flags, supported by comprehensive guardrails spanning human judgment, security validation, and policy enforcement. Organizations that successfully implement these practices will realize the full potential of AI coding acceleration while maintaining system reliability and security. The key is recognizing that faster coding requires proportionally more sophisticated deployment and validation processes, not just scaling existing manual approaches. ## 🔄 Refactor Appendix — How to build the guardrails your AI releases actually need > Density: 2–5 Literature atoms (H3), 0–3 Permanent candidates (H4). > Defaults: 3 Literature, 1 Permanent. Resonance filter: if you can't > articulate in one line why a claim matters, drop it. ### 📚 Literature Atoms (H3) #### AI code velocity creates deployment bottlenecks without proper guardrails > [!quote] ID:rw_ai_velocity_001 > We're seeing four to 20 times the number of changes being pushed out the door or at least through the process that we were seeing before. And we all know what happens when we have an increased number of things changing at any given time. That opens up the risk that something's going to go wrong. **Context & use**: - Core problem statement around 8:30 in presentation - Why this matters: Identifies the fundamental mismatch between AI coding speed and deployment capability **Tags**: #xref #articleseed **Highlight**: [view highlight](https://www.youtube.com/watch?v=Xt7y6CKDqcI?t=510)  #### Feature flags enable granular rollback without affecting entire deployments > [!quote] ID:rw_feature_flags_002 > If you have a similar problem where you've rolled out a small feature, one of the small features in your deployment to your power users or your premium users and they start having a problem, you can turn that flag off without having to do a roll back. You can just turn off that individual flag. **Context & use**: - Technical explanation around 15:45 showing CD vs feature flag approaches - Why this matters: Solves the blast radius problem when AI generates many features per deployment **Tags**: #xref **Highlight**: [view highlight](https://www.youtube.com/watch?v=Xt7y6CKDqcI?t=945)  #### Three guardrail types required for AI code validation > [!quote] ID:rw_guardrails_003 > What are three types of guardrails we can put up? Well, one, we can put up human judgment. Also, we need some sort of security evidence. And then you need policy enforcement because AI is really good about coming up with its own workarounds for things. **Context & use**: - Framework presentation around 22:10 defining comprehensive guardrail strategy - Why this matters: Provides actionable taxonomy for implementing AI code safety measures **Tags**: #articleseed #xref **Highlight**: [view highlight](https://www.youtube.com/watch?v=Xt7y6CKDqcI?t=1330)  ### 🧠 Permanent Candidates (H4) #### AI-generated feature velocity requires proportional deployment sophistication **Claim** (one sentence, Courtney's voice): Organizations achieving 4-20x coding velocity through AI must implement equally sophisticated deployment and validation processes to realize benefits safely. **Evidence pointer**: [[#AI code velocity creates deployment bottlenecks without proper guardrails]] **Scope & conditions**: Applies to teams using AI coding tools extensively; breaks down when deployment infrastructure cannot scale with development output **Links**: - [[DevOps transformation patterns]] — because extends existing CD practices - [[Technical debt management]] — because applies-in high-velocity development contexts - [[Risk management frameworks]] — because contrasts with traditional deployment risk models  ### ✏️ Fleeting Queue - Progressive rollout percentages (1% → segments → full) need optimization research for different feature types #fleeting #inbox - Integration patterns between WordPress deployment tools and feature flag platforms worth exploring #fleeting #inbox - Policy-as-code examples for AI coding standards could be valuable community resource #fleeting #inbox ### → Things 3 (max 2 actions) - [ ] Research feature flag integration options for WordPress core development workflow #things - [ ] Evaluate automated cleanup tools for managing technical debt in AI-assisted projects #things --- # 📄 Full Transcript *Auto-generated from YouTube captions · timestamps roughly every 30 seconds · [watch on YouTube](https://www.youtube.com/watch?v=Xt7y6CKDqcI)* **[00:12]** Well, hey everybody. Welcome. We are going to talk about how to build the guard rails your AI releases actually need. Welcome and thanks for joining. My name is Eric Nukem. I'll do a little intro in a minute, but uh please note that we will have be taking questions throughout. So if you have a question comes up randomly at a given section, you know, don't hold back. Go ahead and open that question box in the tool and uh go ahead and fill it out anytime you want. Have it ready so you can be asking those questions and I will get to those **[00:43]** at the end if I don't see them as I'm going through here. So uh please do that. Anyway, my name is Aaron Nukem. I work for Harness. I work on our feature management and experimentation uh product. That product came from Split. You might remember Split a while back was acquired about a year a little over a year ago was acquired by harness and we've now integrated that into our platform and I'm going to show you some ways that that integration is uh luckily helping to deal with this problem of the AI coded uh feature velocity coming our **[01:15]** way. So let's go ahead and get right into it and I'll tell you a little bit more about uh what we're going to be talking about today. So, I've done, you know, kind of the introduction here. Um, but we're going to talk about some AI implications. I think this will resonate with most people. Every customer I talk to is dealing with this problem. So, I think this will be really good for uh for us to talk about today. And then I'll give you a little bit of an overview of harness. Not really going into too much detail, but I just Harness does so much and I want to make sure we're focused on just the elements that I'm going to be uh focusing on today. Uh **[01:49]** I'll also give you a little bit of an overview of uh continuous delivery and feature flags, what the differences are, how they come together, and then of course how they come together to solve this particular problem. We'll talk a lot about guardrails and validation. I don't want to give that one away too much, but those are the two areas we're going to focus on to kind of fix some of the some of the issues we're having right now due to AI generated code. And then I'll give you some resources and of course we'll at the end there'll be time for a few more questions. So, like I said, don't hesitate. Don't wait till **[02:19]** the end to ask them. Go ahead and ask your questions as we go through here. That's the best way to do it. And I think we'll also have a poll here at some point to try to see where everyone is at on this spectrum of how they're dealing with this problem, too. So, let's just go ahead and get into it. So, when I talk about uh you know, the problems with AIdriven code workflows, what am I talking about? Well, in talking to customers and I think industrywide, even if you just read the news around what's going on with AI, I think it's clear that people are okay **[02:51]** with the coding and some of the review part of the process. So, people are using Claude, people are using C-Pilot, people are using cursor, they're using chatbt. they can produce code quickly and companies have been very willing to give people the uh the tools that they need. They're not holding back on any of this, right? Uh in fact, most people are encouraging people to use the tools that they are uh uh that they that they want, **[03:23]** right? They're not it's it's like no, you need to use more tokens. Some people are being measured on that, I think. So, the coding part is pretty well taken care of. people are are um really excited to use those tools. No problem there. Now, what happens? We start developing more and more code. We start creating more and more features that we want to release. Things are going great. Things are looking great. But then we get to the point where we need to roll these things out. And what happens to the size of our deployments? They explode, right? So, you may have had a deployment in the past with, I **[03:53]** don't know, five, 10 features or something like that. We're seeing four to 20 times the number of changes being pushed out the door or at least through the process that we were seeing before. And we all know what happens when we we we have an increased number of things changing at any given time. That opens up the risk that something's going to go wrong. So, we have to start focusing on testing and deployment a bit more. And we can turn some of that over to AI. But as I'll show you in a minute, there are some easy ways to also mitigate some of **[04:24]** the problems that happen when you do that. And then uh last but not least, we have release and experimentation. And this last part of the puzzle is where you actually want to use something like feature flags or an experimentation tool to validate that those changes that you have made with AI are actually doing what they're supposed to be doing and especially not causing a problem with either a security vulnerability or a um an impact to your user experience. **[04:54]** Right? So that seems to be where we we want to concentrate on improving at this point. And why do I say that? Well, uh we actually did a survey recently and we t we asked um real users about how they're dealing with this problem and we got back some really interesting results and and hang on till the end because I'm going to give you a link that you can use to download this particular report and read all the findings, not just the highlighted ones I've pulled out here. So in that report, what did we find? Well, we found that, **[05:25]** you know, a lot of people still need human review for every single line of AI generated code. And if you're writing a lot more code, but you still need a human to review every line of it, that's not very efficient. And it might look like your your code development engine is running great, but you're still not getting those features out into the wild um at the rate that that uh you would like to. Uh we also found that less than half of people have put in uh AI guard rails for **[05:56]** AI generated code. Um this could be a real problem. We've all seen stories in the news where somebody didn't have a guardrail in place and they deleted their codebase or they deleted customer orders or what have you and uh the AI just went ahead and did it and you know was like oh sorry you know my bad you know and that's not good enough in this day and age. Um now what we did see was we saw that 58% of people are running more experiments. This is a good thing uh in one way which is they're trying to keep up with um running more experiments **[06:28]** because they have more features. But the problem is their systems may not be able to really support all those extra running experiments because they're not efficient enough or they're not geared to that volume. we really should be running experiments on every feature that we release or that should be the exception is when we don't do it right. Um, and last but not least, only 36% said AI had improved the quality of their software. Again, sort of similar to the first point in that you need to go back and actually review things or the results aren't quite up to what you **[06:59]** thought they would be, right? You you thought that the quality of software would be great. Turns out there's bugs that are getting into the mix and hopefully you're catching those either in test or behind a feature flag so you can turn those off quickly. So, a lot of really interesting results. Again, I'll give you the link for this when we get to the resources at the end or they'll be in the resources of your uh when you uh after the webinar when you get the resources in your email. Um, okay. Well, let's take a I'm really curious to know how everyone on the call is doing with this. And so let's just do a quick poll **[07:32]** and see what you're doing. It should have popped up on your screen now. And the poll says, "How mature is your team's use of AI in the SDLC?" So go ahead and fill out that poll. The choices are, you know, you're not using it all. Maybe you're using it for, you know, a few people are experimenting with AI. Maybe you're using it, but it's supervised. Maybe AI is uh integrated with guard rails. And maybe no humans are involved anymore. So go ahead and vote on those options and let me know where you're at in your journey here, **[08:04]** your AI journey in deploying and releasing new features with AI. While we're doing that, I can tell you that in talking to a few folks, uh we found out that um some people have gone a little too fast here. So, uh, I was talking to one Fortune 500 company recently and they they gave, you know, everybody all the tools they need and they didn't put in any guardrails and **[08:35]** what they found were that that people were so used to using AI. I'm sure many of you have felt have seen this. I I have myself where you you start to trust the AI maybe a little too much and they had bugs creeping in and because the demand was so high on the developers, they weren't actually reviewing anything and they were just hitting, "Yep, looks okay. go ahead and uh do that build or or commit that code and uh bugs were creeping in so they had to go back to a much more manual code review process. Um and so we are seeing that as well. **[09:07]** Okay. Well, let's just go ahead and take a look at the results of the poll. So, you know, it looks like most everybody is right in the middle here. Uh AI is used but not supervised. So, people are still struggling with this problem. you know, they're they're trying to use AI in the SDLC, but uh they're not a actually able to to get all the way to, you know, the Nirvana, I guess, if you want to say, where no humans are involved anymore. But yeah, about 43% of you are right in the middle where you're using AI, but still very much **[09:38]** supervised. Um, you know, a good portion of you, about a little less than a third are actually using AI with integrated guardrails and then the rest are still experimenting. So, uh, very interesting and very much on par with the other times that I've asked these questions. Okay, so let's go ahead and move on to how do we solve the problem, right? That's what the bulk of this is going to be about. So, uh, we're not going to focus on the coding problem because you all are really good about giving tools to your developers to code with. Let's **[10:08]** focus on the testing and deployment and release and experimentation phases of doing your uh um of releasing new features, right? And so I thought we would tackle this in two different ways, right? For the testing and deployment and release and experimentation uh phases, we're going to be talking about uh how to set guard rails, what kind of guard rails are should be in place, and then also how to validate when you've actually released a feature. So validation is super important here. I've talked to uh some companies that that **[10:40]** basically have uh I won't say dictated because they haven't quite gone that far, but what they've said is the goal, what they're measuring against is whether they're running an experiment on every single feature release. Super important. And also as part of that validation comes what I calling for now codification or that last part of the process where you need to go in and clean up your code. So, because I work with feature flags, I'm going to show you an example of how you can go clean up feature flag code as well. **[11:11]** All right, before I get into those two things, I do want to just set the stage here, uh, and tell you a little bit about what I'm going to be using my, you know, obviously I come from harness, so I'm going to be talking about our tools here. You may have other tools, that's totally fine. Maybe you've got a DIY approach, whatever applies to you, but I just want to let you know my context for this. So um Harness has built basically an AI um foundation if you will with the Gentic Workflows software delivery knowledge graph in the middle that pulls **[11:42]** in information from different areas so it has context and then also intelligent workflow orchestration. Um and that results in not just feature flags but a bunch of different products that we sell. Uh we have products for velocity, we have products for reliability, we have products for security, and we have products for efficiency, including things like cloud cost management. And so we're not going to be talking about all of this today. We're really going to only be talking about continuous delivery and feature feature management **[12:13]** and experimentation, although I'll throw a little bit a little plug in there for security, too, because we can never forget that. But th those are the two areas I'm going to be focusing on today. All right. So, let's talk about a little bit about continuous delivery and then I'll talk a little bit about feature flags in a minute. Um, but I want to make sure that people understand the difference uh between the two tools and how they apply here, right? So, with continuous delivery, I mean, that's pretty much what people know harness for. That's our bread and butter. That's where we started. Um and uh with **[12:46]** continuous delivery obviously you you build your software, you test your software and you deploy your software and you deploy it you know as one big deployment and that deployment could be an update to your application and it has a number of features underneath right so you've got all your features baked into that deployment you throw it over the fence and then when it comes time to do uh deployment testing you know maybe you do a canary deployment where you can actually deploy it to a small subset of your users you do your canary you see how things are going and then at some point when things look good you can **[13:18]** uh basically uh deploy to everybody right so that's kind of the typical way things work with CD however if something goes wrong with a CD deployment what happens is you have to do a roll back now harness has tools to make automatic roll backs very easy and and and painless at least compared to doing them manually but it still means rolling back all those features that you just put out so whoever's been exposed to those features that tend 20% or something is going to notice maybe a little bit of a difference um across even the features that weren't affected **[13:49]** by wherever this problem took place. This problem may may have taken place downstream in a in a small feature somewhere that was causing a security vulnerability, but you had to roll back everything, right? And uh take those take those features kind of away for a little while until you can fix things and roll them out again. Um it's great to have automated tools for that, but it is kind of can be kind of painful to go through that process. Um, with feature flags, we can do things a little bit differently. So, we can still do our deployment, our big deployment, uh, but if you bake feature flags in, then you **[14:21]** can treat each one of these features as an if else statement in your code. And if the feature flag is turned on, then you can you can uh roll this out to a very small subset. Most people start at just 1%. I know I say 10 and 20 to make things nice round numbers here, but typically people will roll this out to just 1% of your of your segment that you're targeting. Maybe it's a particular region, maybe it's power users, maybe uh uh uh who knows, right, what your segment are region or or or something like that or maybe a platform **[14:52]** that you're rolling this out to. And you can do this one at a time. And so what happens is if you have a similar problem where you've rolled out a small feature, one of the small features in your deployment to your power users or your premium users and they start having a problem, you can turn that flag off without having to do a roll back. You can just turn off that individual flag. The rest of your deployment is unaffected. Um, and you've essentially rolled back instantly to the previous state of your code because of the way this works. So uh, feature flags can do **[15:24]** all that. Now what feature flags can't do is they can't redeploy your entire application. Um they can't roll back that that entire update out all at one time for you. But they can mitigate the risk of a small subset of that deployment causing a problem where you would have to roll everything back. So the best of both worlds here are to use both, right? You use CD to go ahead and do those deployments and then you use feature flags to one at a time uh uh deal with each one of these features. Now why is this important? Why am I talking about all this? Well, it's **[15:55]** because of that first problem I talked about before where you have, you know, 20 times the amount of features that you have to release at any one given time. So, we're trying to solve that problem of things going wrong in that huge amount of features, right, that we have now. Not just one, not two, but maybe 20 things, maybe a hundred things that we rolled out at one go. And we need to be able to have a way to deal with the increased amount of exposure that we have so that when something goes wrong, we don't have to roll all of that back. We can just roll one at a time. And that **[16:26]** certainly helps us mitigate that risk to a certain extent. Um, so by combining these two things, it's really the best of both worlds. Now, let me give you a more practical example. Maybe you have a mobile app, right? This is example of a banking app. My the my bank. Uh, who who else banks at my bank? I open all my accounts there. Um, but you can imagine there's all this competition out there, right? You can see some of the logos over there on the right. Everyone is vying for eyeballs. Everyone is trying to come up with a feature that will get uh their customers **[16:57]** or their potential customers to open more bank accounts or use their app more. And that's another great example of how you can use feature flags and deployments together as as one combined way to deal with this AI code velocity problem. As I as I mentioned in in in the process of doing one single deployment, you may have 180 features kind of tucked behind there that you're turning on one at a time. You're probably experimenting with a subset of those. Although again, I would argue you should be experimenting with almost all **[17:27]** of them uh to know what's actually impacting the bottom line. And then when you uh once those features are validated, you've run your experiments, you know that they work, then the banking app the next time it gets a deployment to the next version, all those things are codified in the code and they become standard features, not behind a feature flag anymore. Um, and that's the way that a lot of people use these tools. Okay, so let's now talk about pipelines real quick. Pipelines is uh people have **[17:59]** deployment pipelines. It's kind of a generic term, but at harness it means something a little bit more. We have our own pipelines feature of the product. The nice thing about it is it spans uh uh multiple different products that we sell, right? It's not just one tool for one product. It actually integrates, for example, with feature flags. I'll show you an example of that in a minute. What this allows you to do is it allows you to very easily, you can set this up with YAML if you want. uh you can very easily deploy pipelines and uh do things in an **[18:33]** automated fashion with guardrails and I'll give you some examples of that in a minute. So you can see here we're doing a verifi manual verification check which uh looks like it failed in this case in the graphic right so we're doing some sort of verification maybe it maybe there was a security problem or something like that some sort of verification happened um and that was uh to protect us against releasing code that was generated by by AI but may have a problem with it um and we don't want to release that into production yet. So it's a great time to do these checks. It's a great time to have these checks in your pipeline. So we'll be talking a **[19:05]** lot about pipelines as we finish out the presentation today. Okay. So we talked about guardrails and validation. I'm going to talk about those separately. So what are the guardrails? What do guardrails even mean? What are we talking about? Well, my definition for guardrails are the controls that decide whether AI generated code and AI assisted changes are allowed to advance. So you you're going to be making these things, but the question is at what point do we do a hard check to know if things are going right or if things are going wrong? **[19:36]** Because nobody wants to end up in the news for a security vulnerability that leaked out, right? Okay. Well, what are three ways we can deal with that? What are three three types of guardrails we can put up? Well, one, we can put up human judgment. That would be kind of a human in the loop problem where we at least we might not need humans evaluating every line of code whether it's correct or not. But at least we need some human checking to make sure that this pull request for example meets our standards or isn't going to break something before it goes into production. Also, we need some sort of security evidence. So, **[20:07]** we need to do some sort of test. And this also gets down to a little bit of regulatory environments as well, right? You need an audit trail, but we'll talk about application security mostly here. You need some sort of uh security check to make sure that you don't have a vulnerability. And then you need policy enforcement because AI is really good about coming up with its own workarounds for for things. You know this if you've used it and it'll say, "Well, I I just created my own naming scheme for this thing, right?" Well, no. If you're working at a company with a thousand people, you need a naming scheme that's **[20:39]** consistent across things so that the incident team knows what's going on if that there's a problem or or whatever. So, you need some sort of way to enforce policies. So, how do we do that? Well, I'm going to tell you how we do it at harness. You can um think about your own products that you use when you're thinking about this, but let's talk about policy as code first and then we'll work backwards. So, as I said, uh you need some sort of policy enforcement. The way that we do it at harness is with OPA style policy. So, we actually have a a policy um uh **[21:10]** when you go in to create a policy, we have examples that you can use and we show you the the code. I always forget what the name of the OPA standard is for writing policies, but we use that same standard to make it easy for people. So, you don't have to worry about if you want to take these policies to somewhere else later, they'll still be valid. Um, and you can actually test them. You can go in here and if you set up a policy, let's use the same example where I have a naming convention that needs to be standardized on and I'm setting up that policy. You can actually go in and test put in a bad policy and a **[21:41]** good policy and see if the policy enforcement works. And here's an example of that enforcement. So, we're running our pipeline. Uh we're running the create feature flag step in our pipeline, which happens automatically as the pipeline progresses. And we get to this point and it says, "Hey, you've got a bad name in here. Um, you've used the name Ola, which doesn't require a Jira ticket number." So, in this case, the company wanted to have the Jira ticket ticket number when as part of the feature flag name. Again, I think this is probably **[22:12]** for, let's say, this goes to an incident team. They need to be able to turn this off because something happened. You want to have that Jira ticket uh available to you. And so, why not just put it right in the name of the flag? Makes a lot of sense. and you can enforce that across your entire organization with policies. So that's the first one. The second one is human in the loop. How do we do that or when do we use it, right? Um well, you want to use it for approvals for production. And some people are going right to production to run certain tests, right? Especially with their user **[22:43]** interface, uh with their user experiment, uh experience, they want to do that in production. But before something goes to production, you certainly want to have at least somebody that is familiar with the code to go through and make sure that it's not going to break something else, right? What about high-risk services, financial transactions or uh regulated data um that uh uh needs to be handled differently? Um unusual deployment patterns. So you you know, you've all been through this where you're starting up a new service maybe in the cloud or something, you haven't done it before. you definitely want to have somebody **[23:15]** validating that to make sure that things are right there as well. And then of course, if you get to some sort of failed verification that I showed you before, right? We're trying to validate u a featured, run a security check, something fails, you want to be able to have a human go in and investigate what's going on. Uh so what do you use for that? How do you get notified that something's wrong? Well, if you're using a product like Harness that we've built in tools to basically integrate with whatever you're using. So, you can obviously be notified. This is an example of an inapp notification where someone needs to go **[23:46]** into harness and put in yes, I validated that each feature is working, click approve, everything looks good, and then the uh the pipeline will automatically continue if they hit approve at this point. Uh but also you need to integrate with your existing ticketing tools or your incident systems that you have your service nows etc your pager duties and messaging tools like Slack. Those are all ways that people use uh to make sure they're notified so that not only is there a stop an automated stop in your pipeline to say I need to go validate **[24:18]** this particular part uh but you're notified that that you need to go do that in whatever system works for you. So uh really important to keep those in place and we can't forget security testing. So we have application security testing tools um that can go out and again be integrated into these pipelines along with everything else. Uh really important here is dduping vulnerabilities. So one thing that we're seeing with AI code is if there is a vulnerability and you commit that to a centralized repo as part of a **[24:50]** instruction set or something that's going to reproduce that vulnerability over and over again those vulnerabilities can get propagated. So by having a security testing tool like we deliver at harness you can actually get in front of that and and uh prevent those uh duplication of vulnerabilities uh by AI which is super easy to do. Um and then we also have remediation. So, you know, enforcing that governance with those same OPA policies we talked about before, being able to track issues by integrating into Jira and Slack, for example. Same sort of notification stuff **[25:22]** I was talking about before applies here. So, you get notified immediately. You can go in and say, "Hey, uh, you can ask our AI system, for example, I, you know, I saw that there was a problem. What's going on?" It can come back and and pinpoint the problem for you so that you can uh, uh, remediate it quickly. Super important to have this as a guardrail when you're rolling out AI generated code. Now, let's talk a little bit about validation. Again, what do I mean by validation? Well, what I'm talking about here is the process of proving that a **[25:52]** release is safe, useful, worth keeping, it's important, and codifying. Again, if you have a better word for that, let me know. I don't know what it is. Validating, codifying. Basically, you need to go in and and and um fix your code or or uh codify your code to the point where you've taken the feature flag out. The process that you validated is working and is important for your business is now in the next release, right? It's in the next deployment. So, what are we talking about there? What kinds of things fall under validation? Well, for me, it's validating the **[26:23]** release process itself. It's integrating this into the release process so that you can keep things moving quickly and you don't get bottlenecked with all of this AIdriven code that's coming your way. Experimentation again got to make sure that this stuff is impacting the bottom line or affecting your users in the way that you want. And then obviously tech debt is a big deal. Tech debt can increase uh with right along with the number of changes that you're putting out uh due to AIdriven code velocity that tech debt is also accelerating uh because of that as well. **[26:54]** So if you're not taking steps to clean it up, you're going to run into problems. So let's first start about how we can deal with that release process and feature flags. I gave you all the examples of this. I'm not going to go into too much more detail here, but basically you can think of it as uh um you know the last bullet here, limiting the blast radius, right? So, one of the things that you want to do as you're releasing more and more features is to make sure that the impact, whatever impact you have, if there is a problem, **[27:25]** that it's limited. It's limited to maybe your internal users only. Wouldn't that be great? You release this to internal users. Have them take a look at it first. It's not going to catch everything, but it's certainly not going to land you in the newspaper or on uh uh your your your daily Google newsfeed, right? So, not only that, but you can roll out progressively by user, by segment, by region. As I mentioned before, doing a progressive roll out is much safer um and can be much more effective than rolling out to everybody all at once. And then, of course, **[27:55]** instant kill switches. That's a great way to look at feature flags. This whole area of feature flags has developed. used to be really for, you know, people that were wanted to open up a beta, for example, to some uh customers and they would turn on the feature flag and the and the the customer in in question would get the the beta of a new feature or new new uh uh product or something like that, right? It's evolved u much more since then and now it's used to do all sorts of things and especially to guard against problems when you're doing **[28:26]** these AI a lot more AI generated releases like this talk about experimentation as well is there a bottleneck with experimentation well what we found is that for advanced experimentation teams uh even if they've done everything right so they've built up their experimentation team they've created dashboards, they're doing dynamic configs, they're uh plugged into warehouse native, they've shared this learning experience across their team, they still hit a bottleneck. And the **[28:58]** problem is exactly what we've been talking about. They build it to a point where maybe they're doing experiments weekly, but they can't get beyond that because they just can't run any more experiments any faster. And what we found the bottleneck is here is actually the uh same thing that we've been talking about which is they aren't building the experimentation platform into the rest of their deployment pipeline. So, uh, if you're using a separate experimentation tool, that's fine. Just know that you may run into **[29:29]** this bottleneck problem where you hit a certain ceiling and you're not able to keep up with the code velocity that's that's coming your way, especially if you're changing your expectation with experimentation to be experimenting on everything, right? Think about your experimentation teams here, right? they they they were maybe running, you know, 10 experiments or something like that, you know, now they're running a hundred experiments. How are they going to keep up with that? How are they going to automate that? Um, it can be a real problem. And and the best answer that we found is to make **[30:01]** sure that the tools are integrated. Uh, the experimentation tool, your feature flagging tool, your CI/CD tool, make sure all of those are integrated. And that will take out the bulk of the of the bottleneck because you can automate all the way through that whole pipeline. Uh, now let's talk about removing tech debt. So, again, I'm going to be focusing on feature flags here because that's my bread and butter. The point is you need to be doing this uh because it's going to come back and **[30:31]** haunt you. Now, what we've done for feature flags is we've actually released an an automated uh feature flag cleanup tool that will go out and it will based on your guidelines, it will go out and automatically find feature flags that are let's say in this case 100% rolled out, right? So, you don't want to be removing feature flags that haven't been fully released yet. So, as you're ramping up, you don't want to be release getting rid of that feature flag code. But what you can do is once it's been uh fully rolled out, the experiment is **[31:01]** finished, you've proven the value of that particular feature, you can have this tool go ahead and run and it will actually go in and identify in your code where those feature flags are and set up uh automated ways to actually uh uh pull that code out and including, you know, doing a uh uh uh a PR for you on your behalf and things like that. We're always improving this feature. It's it's uh hot off the press, if you will. So, we're getting a lot of feedback on it and a lot of good feedback. And uh this **[31:33]** is something that we feel can really help people go in and clean up their code because the last thing you want to have is old code hanging out in your co codebase that an AI tool can reference and perhaps make assumptions on your behalf that you don't want it making. So, cleaning up this code is really important. Okay, let me give you an example of a company that's seen dramatic results by uh following the approach that I just laid out, right? Especially with the integrated pipelines, being able to automate things, having enough review in the loop, uh human review and and **[32:06]** guardrails and things like that that they can move very very quickly. So, this is Speedway Motors. We've actually uh done another lead dev webinar with them. I'll give you a link to that in just a minute. Um, but before they started using our tools, they were only releasing a couple times a month. That's as fast as they could do it. And because they weren't didn't really have the uh the right tools in place to run experiments to know what was impacting their e-commerce tools or the e-commerce uh product that they have, they weren't really able to see any increase in conversion rates when they did do **[32:37]** testing or when they did release new features. So once they started using our tools, they found very quickly that they got immediate ROI because they were able to go from releasing two times a month to releasing multiple times per day um sometimes two, three times per day. So their deployment frequency went way up. They were also able to see an immediate impact and they were able to make judgments on whether new features were helping or hurting and then move forward very quickly and get those uh changes in their codebase and get them rolled out **[33:07]** uh in the next deployment which was really important. Um and they found that there was a big impact to the bottom line uh because they had the visibility that they needed because they didn't have separate tools getting in the way of each other and a lot of delay. they were actually able to iterate quickly and say, "Hey, this particular special that we ran on the website or the way that we did this is actually causing people to order more stuff and so we need to do that again and again, you know, every year seasonally, whatever it is, whatever the problem is." But they were able to to validate that that was working, which is really great. **[33:39]** Okay, let me go ahead and give you some resources now, things that you can take with you. Uh, these will be offered to you as well after the webinar, but I want to talk about a few things. So, coming up on June 17th is uh our annual flagship event that we do at Harness. Uh Split used to do this before we acquired them and we've kept it going and it's special because this isn't just us talking to you about what we think. What we do is we actually go out and get actual practitioners and have them come in, practitioners that are doing incredible things and they talk about **[34:11]** the problems that they're facing and they talk about how they're solving them. Uh whether it's with harness or not, it doesn't matter. We have some folks coming in from Curser to talk about the AI revolution and what that means. We've got uh some folks coming in from some financial institutions, some retail companies. They're coming in to talk about how they're dealing with this AI velocity problem, what they've put in place, how they're running experiments, how they're cleaning up their code. Um all of that is going to get talked about by practitioners. So, uh you can go out and sign up for this. Just do a Google **[34:41]** search for Harness Flagship 2026 should come up and you can register for that event. And be sure to register even if you can't attend the whole thing. It's free. It's virtual. Um you can listen to it in the background, but if you get pulled away or whatever, you'll have the replays available to come back and listen to as well. I don't think you're going to want to miss it. And uh definitely go out and register for that event. Um I also mentioned this report that we put out along with lead dev. So, there'll be a link to download that report. I I recommend going out and **[35:13]** doing that and kind of seeing where you fit in. I can tell by your poll responses that uh you're going to fit right in with the responses on this survey. But there are some interesting takeaways. Um, and we also talked about this in a little bit more detail on a webinar called d-risking releases in the AI era. And so, I think I was on that one. And, uh, we talked about what the implications are. We kind of drilled into some of the details, some of the findings from that report in a webinar fashion. So again, great listening material. If you're working or doing something around the house and you want **[35:44]** to hear the the output of that report and what the implications are, highly recommend that one. And I talked about Speedway Motors. We also have uh that webinar. You can go back and listen to them. Again, practitioners, these are the people from Speedway Motors that implemented these things. Definitely go out and take a look if you're interested. One of the things they talked about which I did not realize until we were on that webinar talking about it was um they even control some of their physical uh resources through feature flags. So I think the example was they control their warehouses. If **[36:15]** there's inclement weather and the warehouses shouldn't be running, they actually flip a feature flag to turn off the warehouse, which is uh which is amazing to me. but it's an easy way for them to control some of those things uh phys in the physical world as well as in their application. Um, and last but not least, uh, if you want a demo of any of these things, if any of these topics that we talked about through guardrails and validation, if any of those are problems for you right now and you just want a a a demo of how **[36:46]** it would work, that's more specific to your particular use case or the problem that you're facing, uh, you can scan this QR code or just, uh, go out to the website and sign up for a demo. Highly recommend you do that. There's no cost, no risk. We'll just ask you a few questions. will get on the phone and we'll walk through the product in a way that's more applicable to you. So, go sign up for that demo and uh I'd love to see some of the folks on this call actually uh in the future perhaps on a demo call. That would be great. All right, **[37:17]** so for now, uh let's go ahead and get over to some questions. See if I can bring up my Q&A here. Uh this is the time if you do have any questions uh go ahead and put them in now. We'll take a few more minutes. I do have one. Uh do you support rolling back database changes? Um yes we do actually. We can help with uh uh we actually have a tool called um DB DevOps and the DB DB DevOps **[37:49]** tool. I have to be careful because I want to say DevOps. The DB DevOps tool actually can help you rolling back your DB changes or doing schema migrations. um, you know, it's not my particular um, uh, product that I work on, but I do know that we do have it. It does integrate with the pipeline process that I showed you before. So, as part of your process, you're rolling out a new feature, you need to move someone over or do some sort of a database uh, change, you can actually build that into the pipeline and have it happen **[38:20]** automatically. Um, I'll give it just one more second to see if there's more questions that came come in. One of the interesting ones that came in uh last time when I was talking about this was how do I convince my management that this part of the process is important. Um, that that can be a struggle right now. I think a lot of people are focusing on the code development side of things and how many tokens you're using and how much code you're pushing out. **[38:52]** Um, but they're starting to realize that that code doesn't always make it all the way out into uh production because of this deployment problem that we've been talking about and the release and and experimentation problem that happens down the line. Um, it's not just that you're developing more code, it's that you're doing more changes and you need to be running experiments on more stuff. So, if you don't have a good tool, uh especially if you're running at a large organization with thousands of developers, if you don't have a good tool to help you with the rest of the process, um you could be in for a world **[39:25]** of of hurt. And uh you might not get the results that the executives want to see. You might get the results that your manager wants to see, but the executives might not be too pleased that there's all these changes backed up in the pipeline. Um let's just see if there's anything else. Uh let's see. How do you put in guardrails for uh safe handling PII or other security guardrails while pushing changes? This is a good good question. I talked about application security and we **[39:57]** talked a little bit about regul regulated environments. Um you can there's a couple of things I'll mention here. So you can definitely put in guardrails as part of policies for dealing with PII. That's probably the the first way that I would handle that is put in a policy that says, you know, this this shouldn't this database shouldn't be accessed or this uh type of information shouldn't be shared. This uh credit card uh sequence of numbers shouldn't be should never be used or there's policies that we have in place for for all of those things. Um and of **[40:28]** course, every company's going to be a little different. So, they're flexible enough you can create your own policy to make sure that that's not happening. Um flexibility is key. That's why we offer, you know, YAML, you can do pipelines as code, for example. Um, and and so, yeah, it's very flexible. Putting in those policies for that information that shouldn't be shared is not difficult, and it definitely should be part of a check as some at some point and as part of your uh pipeline for sure. **[40:59]** Um, just looking at the questions here, one more. How do you navigate a testing team who wants to test every permutation of a feature flag? Basically, if we have multiple feature flags in flight, we need to ensure tuning them uh turning them off is going to work just because it's coded behind if else there are still things that can go wrong. Uh, I'm going to have to interpret this one a little bit. Um, **[41:32]** if you you you need a tool that supports being able to test every permutation. Um, it's so obviously there's going to have to be a conversation about how much testing can take place. Um, I'm assuming this is a velocity question, right? Like, oh, we haven't tested uh this particular use case or this particular thing, so we can't move forward yet. Um, you can run multiple tests at once. You can have multiple feature flags turned on. You can have multiple segments turned on. You can have multiple treatments turned on. And you can be doing multiple experimentation turned **[42:02]** on. There was a demo I did the other day where we showed all the different permutations that we had running at the same time. And I think it was 256. I don't know. I don't know if there's an upper limit, but there was 256 different things being tested. And then automatically we were able to decide which one of those was working and which ones weren't. And then we were able to progress that roll out very quickly and determine what the best one that basically won the evaluation contest um so that we could roll that out more broadly. So I know that we can do it at harness. Um hopefully your experimentation tool is able to do that **[42:34]** as well and hopefully it's integrated with the rest of your pipeline to make that process easy. Um okay, I think we're about out of time and I think that's all the questions we have for today. So thanks everybody for joining. Thanks for spending a little bit of time with me. Really appreciate it. I hope I get to talk to you on one of these in the future and I hope you have a great day. Thanks a lot and we'll see you later. Bye-bye.