Case Study Feature Flags IRL - CourtneyR.dev's 2nd Brain

Case Study: Feature Flags IRL ![rw-book-cover](https://img.youtube.com/vi/RAUscXUlZmI/maxresdefault.jpg) 🔗 **Source:** [Case Study: Feature Flags IRL](https://www.youtube.com/watch?v=RAUscXUlZmI) > [!note]+ Document Notes > tags: #feature-flags #progressive-delivery #experimentation #deployment-safety #continuous-integration > author: Lead Dev > source: YouTube > reference: https://www.youtube.com/watch?v=RAUscXUlZmI > date created: 2024-12-19 15:30 > > # 🏷️ Title: Case Study: Feature Flags IRL >  ## 🧰 Action Inbox - [H1] Implement safety feature flags for immediate rollback capability — avoid two-week sprint rollback nightmares #deployment-safety - [H2] Create progressive rollout strategy starting with internal team targeting — test in production safely on ourselves first #testing-strategy - [H3] Set up automated feature flag debt monitoring with API alerts — prevent zombie flags accumulating #technical-debt ## 🧭 Table of Contents - [[#📝 Summary]] - [[#📖 Main Content]] - [[#🗒️ Notes]] - [[#📚 References]] - [[#🗣️ Paraphrase]] - [[#📰 Newsletter]] - [[#✅ Action Items]] - [[#🧪 Hypotheses]] - [[#📊 Methodology]] - [[#❓ Questions]] - [[#🔮 Future Research]] - [[#💡 Implications]] - [[#🧠 Learning Styles Assessment]] - [[#📊 Accessibility Assessment]] - [[#🌍 DEIB Assessment]] - [[#✅ Logical Validity]] - [[#🛡️ DARVO Analysis]] - [[#🎯 Conclusion]] ### 🔗 References - [Speedway Motors Case Study](https://harness.io/customers/speedway-motors) - [Lead Dev Slack Community](https://leaddev.com/slack) ## 📝 Summary - Speedway Motors transformed from risky fortnightly releases to safe multiple daily deployments using feature flags - The transition addressed blast radius reduction and progressive delivery without hiring new staff - Feature flags serve dual purposes: safety rollbacks for developers and business experimentation for stakeholders - Implementation includes both ephemeral experiment flags and evergreen operational toggles (warehouse weather shutdowns) - Success requires managing feature flag debt through automated monitoring and cleanup processes ## 📖 Main Content Speedway Motors, a tech-enabled e-commerce retailer processing thousands of daily transactions, faced the classic deployment dilemma: fortnightly releases created massive blast radius risk. Their solution through feature flagging demonstrates how progressive delivery transforms both engineering safety and business experimentation capabilities. The transformation addressed multiple pain points. Previously, deployments required 4 AM all-hands ceremonies with complete rollbacks if any feature failed. Teams couldn't isolate which changes drove positive business metrics or quickly address failures. Feature flags enabled granular control, allowing immediate toggles for individual features while maintaining overall system stability. Implementation success stemmed from addressing different stakeholder motivations. Engineers valued safety flags that wrapped new features in toggles, enabling immediate rollbacks without full deployments. Business stakeholders appreciated experimentation capabilities, testing features on small user segments before full rollouts. Both groups benefited from production testing on internal users first. ## 🗒️ Notes #### ⏱️ [00:00](https://youtu.be/RAUscXUlZmI?t=0) — [H1] Speedway's deployment pain points - Key idea: Fortnightly releases created risky all-hands deployment ceremonies - Evidence: "4 o'clock in the morning, everybody on deck, not sure what's going to happen" #### ⏱️ [04:32](https://youtu.be/RAUscXUlZmI?t=272) — [H2] Feature flag definition and evolution - Key idea: Feature flags started as simple toggles but evolved into comprehensive experimentation platform - Evidence: "It's a toggle. You're turning a feature on or off... it's evolved way beyond that" #### ⏱️ [07:15](https://youtu.be/RAUscXUlZmI?t=435) — [H1] Product page redesign success case - Key idea: Major page redesign used progressive rollout with subsequent micro-experiments - Evidence: "We were able to do an experiment on the button color... Orange and green have been measured to perform the most" #### ⏱️ [11:20](https://youtu.be/RAUscXUlZmI?t=680) — [H2] No additional hiring required - Key idea: Existing team adopted feature flagging without technical background requirements - Evidence: "We did not have to hire anybody. Our team picked it up. It was very intuitive" #### ⏱️ [13:45](https://youtu.be/RAUscXUlZmI?t=825) — [H1] Safety vs experimentation motivations - Key idea: Different stakeholder groups found different value propositions in feature flags - Evidence: "Business-minded people really like the experiments... engineers really like the safety" #### ⏱️ [16:30](https://youtu.be/RAUscXUlZmI?t=990) — [H2] Production testing on internal users - Key idea: Teams can deploy to production and test on themselves before customer exposure - Evidence: "We do test in production. We just don't test it on everybody" #### ⏱️ [19:10](https://youtu.be/RAUscXUlZmI?t=1150) — [H1] International shipping failure example - Key idea: Feature flag enabled quick rollback when customs charges revealed UPS integration issue - Evidence: "Turned that feature flag back off... gave us that time to be like, okay, we're stable" #### ⏱️ [22:40](https://youtu.be/RAUscXUlZmI?t=1360) — [H1] Weather-based warehouse shutdown - Key idea: Evergreen feature flag controls warehouse operations during inclement weather - Evidence: "We have a feature flag that controls our warehouse... when we have poor weather and we can't ship out" #### ⏱️ [25:15](https://youtu.be/RAUscXUlZmI?t=1515) — [H2] Complex recommendation system configuration - Key idea: JSON configuration in feature flags enables complex A/B/C/D testing without deployments - Evidence: "We can just go in there and change the title without ever having to do a deployment" #### ⏱️ [28:50](https://youtu.be/RAUscXUlZmI?t=1730) — [H1] Feature flag debt management - Key idea: Automated API monitoring alerts teams to clean up long-running non-evergreen flags - Evidence: "Query harness API... if it's been on for over a month. Are you sure you want to keep it?" #### ⏱️ [32:20](https://youtu.be/RAUscXUlZmI?t=1940) — [H2] Metrics and measurement approach - Key idea: Immediate data collection compares enabled segments to control groups with focus on feature-specific metrics - Evidence: "The closer to the metal that you can get the better... search product engagement... not overall revenue increase" #### ⏱️ [36:45](https://youtu.be/RAUscXUlZmI?t=2205) — [H2] Managing flag dependencies and conflicts - Key idea: Communication and truth tables help avoid conflicts when multiple flags affect same areas - Evidence: "Make a truth table off off on on off on on and figure out which one wins" #### ⏱️ [41:30](https://youtu.be/RAUscXUlZmI?t=2490) — [H1] AI model configuration use case - Key idea: Feature flags enable rapid AI model switching without deployments - Evidence: "Made a feature flag... that just has what model I want to use... so I don't have to do a deployment every time" ## 📚 References - [Speedway Motors Customer Story](https://harness.io/customers/speedway-motors) - [Harness Feature Management](https://harness.io/products/feature-flags) - [Progressive Delivery Patterns](https://progressivedelivery.io/) - [Lead Dev Community](https://leaddev.com/) ## 🗣️ Paraphrase - Speedway Motors eliminated risky fortnightly deployment ceremonies by implementing feature flags that allow safe, frequent releases with immediate rollback capabilities and progressive user targeting - The transformation required no additional hiring as existing teams quickly adopted intuitive tooling that serves both engineering safety needs and business experimentation requirements - Success depends on managing feature flag lifecycle through automated debt monitoring, clear evergreen vs ephemeral distinction, and proper metrics focused on feature-specific rather than broad business outcomes ## 📰 Newsletter **Feature Flags Transform E-commerce Deployment Safety** Speedway Motors' journey from nerve-wracking fortnightly releases to confident daily deployments offers a compelling blueprint for WordPress teams managing release complexity. Their key insight: feature flags serve dual masters effectively. Developers gain safety through instant rollback capabilities, while business stakeholders unlock experimentation potential. The weather-triggered warehouse shutdown example perfectly illustrates how feature flags become operational infrastructure, not just deployment tools. **WordPress Applications:** - Plugin activation/deactivation for A/B testing new features - Theme switching capabilities for progressive design rollouts - Performance optimization toggles for high-traffic periods - Third-party service fallbacks during API outages The "test in production on ourselves first" approach resonates strongly with WordPress staging workflow improvements. Combined with automated flag cleanup monitoring, this creates sustainable development velocity. **Resources:** [WordPress Feature Flags](https://wordpress.org/plugins/search/feature-flags/), [WP-CLI Deployment](https://wp-cli.org/), [Performance Monitoring](https://wordpress.org/plugins/query-monitor/) ## ✅ Action Items #### DevRel - Developer advocacy: Create feature flag best practices content for conference talks and workshops with deployment safety focus - Education: Develop progressive delivery curriculum covering safety flags, experimentation flags, and operational toggles - Developer experience: Document feature flag integration patterns for common development workflows and CI/CD pipelines - Developer marketing: Promote feature flag adoption success stories highlighting reduced deployment stress and faster feedback loops - Community management: Facilitate discussions around feature flag debt management and cleanup automation strategies #### OSPO - Policy & Governance: Establish feature flag lifecycle policies distinguishing ephemeral experiments from evergreen operational controls - License Compliance: Review feature flag tooling licenses and ensure compliance across experimentation platforms - Security Management: Assess feature flag security implications including access controls and configuration management - Software Inventory: Include feature flag configurations in SBOM documentation and dependency tracking - Developer Support: Provide training on feature flag patterns, safety implementations, and measurement strategies - Community Engagement: Share feature flag adoption patterns with upstream projects and contribute to tooling improvements - Strategic Alignment: Define KPIs for deployment safety, experimentation velocity, and technical debt reduction through feature flags #### Community - Maintainers: Support feature flag integration in project roadmaps with safety-first progressive delivery approaches - Contributors: Create first-issue programs around feature flag implementation and documentation improvements - Sponsors: Fund feature flag tooling development and maintenance for open source experimentation platforms - Ecosystem: Coordinate feature flag standards and interoperability across related projects and vendors #### WordPress - Core: Integrate progressive deployment capabilities for WordPress.org plugin and theme directories with feature flag support - Core Performance: Use feature flags for performance optimization rollouts and A/B testing performance improvements - Core AI: Implement feature flags for AI feature experimentation with model switching and prompt configuration capabilities - Mobile: Enable feature flag support in WordPress mobile apps for progressive feature rollouts and user testing - Playground: Integrate feature flag demonstrations and experimentation environments for learning progressive delivery - Community: Develop feature flag best practices documentation and case studies for WordPress-specific deployment patterns - Training: Create lesson plans covering feature flag implementation, safety patterns, and business experimentation approaches - Design: Use feature flags for design system rollouts and user interface experimentation with measurement frameworks - Documentation: Document feature flag patterns for WordPress plugin and theme development workflows - Hosting: Coordinate feature flag support across WordPress hosting providers with standardized implementation approaches #### The WPCC - Programs: Fund fellowships focused on progressive delivery tooling and feature flag implementation for WordPress ecosystem - Projects: Support governance frameworks for feature flag lifecycle management and automated debt cleanup systems - Funding: Provide matched grants for open source feature flag tooling development and WordPress integration projects #### FAIR - Technical independence: Ensure feature flag configurations maintain transparency and avoid vendor lock-in through open standards - Policies: Require funding transparency for feature flag tooling development and experimentation platform choices - Governance artifacts: Create charters covering feature flag decision-making processes and stakeholder involvement frameworks ## 🧪 Hypotheses - Organizations implementing safety-first feature flags before experimentation flags achieve higher adoption rates and lower deployment anxiety compared to experiment-first approaches - Feature flag debt accumulation follows predictable patterns based on team size and release frequency, enabling proactive cleanup automation strategies - Progressive delivery through feature flags reduces mean time to recovery (MTTR) by 80% compared to full deployment rollbacks - Internal-first targeting strategies increase feature quality and reduce customer-facing incidents by enabling production validation before user exposure ### 📊 Methodology - Survey deployment teams before and after feature flag adoption measuring deployment confidence, rollback frequency, and incident response time - Analyze feature flag lifecycle data across multiple organizations to identify debt accumulation patterns and optimal cleanup triggers - Conduct controlled experiments comparing safety flags vs experiment flags as initial adoption entry points measuring long-term usage patterns ### ❓ Questions - How do feature flag adoption patterns differ between monolithic and microservices architectures in terms of safety vs experimentation motivations? - What organizational factors predict successful feature flag debt management vs accumulation of zombie flags over time? - How does feature flag granularity (feature-level vs component-level) impact debugging complexity and incident response effectiveness? ### 🔮 Future Research - AI-powered feature flag configuration optimization based on user behavior patterns and business metric correlations - Automated feature flag cleanup through static code analysis and deployment pipeline integration - Cross-platform feature flag synchronization for multi-environment progressive delivery orchestration ### 💡 Implications - Practice: Feature flags fundamentally change deployment psychology from risk aversion to experimentation embrace, requiring cultural adaptation alongside technical implementation - Policy: Organizations need clear governance frameworks distinguishing temporary safety flags from permanent operational controls to prevent configuration sprawl - Security: Feature flag configurations become critical infrastructure requiring access controls, audit trails, and configuration management disciplines - Education: Development teams require training on both technical implementation and business measurement aspects of feature flags for successful adoption - Funding: Feature flag tooling represents infrastructure investment with ROI through reduced incident costs and increased deployment velocity ### 🧠 Learning Styles Assessment Reading level estimate: 12th grade #### 👀 Visual learners - Present: JSON configuration examples, deployment flow diagrams, metric dashboard screenshots - Improve: Architecture diagrams showing feature flag integration points, visual timelines of progressive rollouts #### 🔊 Aural learners - Present: 45-minute video discussion with multiple expert perspectives and real-world examples - Improve: Audio-only version for podcast consumption, speaker notes for presentation adaptation #### 🗣️ Verbal learners - Present: Clear definitions of progressive delivery, feature flags, and technical debt with automotive analogies - Improve: Story-driven case study format with narrative progression from problem to solution #### 🧩 Kinesthetic learners - Present: Hands-on demo environment mentioned in resources, step-by-step implementation guidance - Improve: Interactive workshop format, guided implementation checklists, sandbox environment #### 🧠 Logical learners - Present: Clear problem-solution structure, metrics-driven decision making, technical implementation details - Improve: Decision trees for flag types, causal diagrams linking flags to business outcomes #### 🧑‍🤝‍🧑 Social learners - Present: Panel discussion format, community Q&A, Slack channel integration for ongoing discussion - Improve: Cohort-based learning programs, peer review of flag implementations #### 🧘‍♀️ Solitary learners - Present: Case study format enabling individual analysis, self-paced video consumption - Improve: Reflection worksheets, self-assessment rubrics for readiness evaluation #### 🌿 Naturalistic learners - Present: Weather-based warehouse shutdown example connecting technology to environmental factors - Improve: Ecosystem thinking about flag lifecycle, environmental impact considerations of experimentation ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading structure, speaker identification, topic transitions - Improve: Timestamp descriptions, speaker role context for audio navigation #### Keyboard navigation - Present: Video player keyboard controls, transcript availability - Improve: Chapter navigation, searchable transcript with timestamp links #### Contrast and readability - Present: Video captions with adequate contrast - Improve: Transcript formatting with proper contrast ratios, large text options #### Text resizing and reflow - Present: Web-based transcript supports zoom functionality - Improve: Responsive transcript layout, mobile-optimized viewing #### Media and captions - Present: Accurate captions throughout 45-minute discussion - Improve: Audio descriptions for visual elements, downloadable caption files #### Forms and interactions - Present: Standard video controls with appropriate labels - Improve: Custom speed controls, chapter selection interface #### --- ## Full Transcript [00:12] Hello everyone and a huge thank you for [00:16] joining us today for today's panel. This [00:18] is a lead dev online engineering and [00:20] action discussion created in partnership [00:22] with harness. Engineering action is [00:25] still a panel but it's kind of a case [00:28] study is probably the best definition of [00:30] it of how things are done in the wild. [00:34] So this discussion will last roughly 45 [00:36] minutes after which both myself and our [00:40] esteemed panelists will head over to the [00:41] lead dev slack to answer some of your [00:43] questions in the panel Q&A and that [00:46] channel is called panel Q&A. And for [00:49] those that are joining us live, you have [00:52] in the resources and I'll drop this link [00:53] in the chat. So you have an invite to [00:55] the lead dev slack. It's a really great [00:58] community of engineering leaders [00:59] supporting and learning from each other. [01:02] So if you give us questions live, we may [01:05] answer them live. we may run out of time [01:07] and continue in the Q&A feature uh the [01:12] in [01:13] Slack or you can just use the Q&A [01:16] feature now in Zoom and we will get to [01:19] your questions either asynchronously or [01:22] right now. So I do welcome everyone [01:24] that's joining us live. That's a benefit [01:25] to joining live but you are deeply [01:27] welcome if you're joining us later. So [01:30] yes, Speedway Motors is a tech enabled [01:34] e-commerce and retail store processing [01:37] thousands of transactions a day. So that [01:40] means that every step in their endto-end [01:43] process from showing up at an ad for [01:45] search results, someone entering a store [01:48] in person or a partner shop all the way [01:52] down to a sale, shopping carts, [01:55] shipping, everything really matters. But [01:57] they were dealing with fortnightly [01:59] releases and you kind of live on edge as [02:03] a developer like is every change a risk [02:06] and how do we reduce that risk? So they [02:09] needed to do something to reduce that [02:11] blast radius, make it safe to [02:13] experiment, what we call progressive [02:15] delivery, which is an umbrella term for [02:18] reducing that blast radius and releasing [02:23] as it says progressively to maybe a few [02:25] of their [02:28] customers or maybe choosing c chosen [02:31] customers, specific ones or potential [02:33] customers, etc. And now with feature [02:37] management and feature flagging, which [02:39] is a specific niche we're talking about [02:41] within the progressive delivery life [02:43] cycle, speedway engineers and business [02:46] stakeholders can release a few times a [02:48] day. And if they want only the first 1% [02:51] of their traffic. So today we're going [02:53] to get both sides of that business tech [02:56] coin with delivery manager Kelly and [02:58] technical engineer Derek. And as a bonus [03:01] questioner and feature flagging fan, [03:03] we're also joined by Harness's Aaron. [03:07] We're going to ground this conversation [03:09] in real features tested, how these [03:12] experiments are set up and managed, [03:14] ephemeral and evergreen future flags, [03:16] and more. So, let's get started with [03:18] introductions. You've probably seen me [03:20] before here. I'm Jennifer Higgins, and [03:22] I'm a freelance technology journalist, [03:25] including with lead dev. And today I'll [03:27] be joined, as we said, by Kelly, Derek, [03:30] and Aaron who are ready to share their [03:32] experience. Can you please introduce [03:34] yourselves? So, Kelly, do you want to [03:36] kick off? [03:37] >> Yeah, absolutely. Hi. Uh, I'm Kelly [03:40] Bear. I'm the senior agile delivery [03:42] manager here at Speedway Motors. I have [03:44] been with Speedway for just over 11 [03:46] years now, so have seen a lot of [03:48] evolution on the tech side of things. [03:51] Um, super excited to talk through things [03:53] today because of that. um just to pass [03:56] along how much we have learned along the [03:58] way and I'm excited to share that [04:00] knowledge. [04:02] >> Great, Derek. [04:03] >> Thank you, Kelly. Uh my name is Derek [04:06] Stapleton. Like Kelly, I also have a [04:09] decade of experience at Speedway Motors. [04:12] Uh my team is primarily responsible for [04:14] customer acquisition and product [04:16] discovery as kind of our niche, but I [04:20] was on the ground floor with the [04:21] adoption of our feature flagging and the [04:24] culture shift that went along with it. [04:26] So I'm very excited to evangelize [04:32] this stuff because it's changed my life. [04:34] I love it. We can get into that a little [04:36] bit more later, but uh Erin, I think you [04:38] might be next. [04:40] >> Yeah, I'll go next. So, my name is Aaron [04:41] Nukem. I'm a senior manager. I work with [04:44] our uh product called FME. It's not uh [04:47] something that a lot of people are aware [04:49] of, but it stands for feature management [04:51] and experimentation. Uh we acquired [04:54] Split.io. So, if you are familiar with [04:56] Split or have used it in the past, [04:57] that's now part of harness and it uh [05:00] integrates with all of our other [05:01] products. So, um yeah, a little bit of a [05:04] little bit of a transition there for [05:06] Split, but we're glad to have them. And [05:08] I will say I've been here for about 6 [05:09] months and the first thing I heard was, [05:11] "Hey, have you heard what Speedway [05:13] Motors is doing? You need to go take a [05:15] look and see what they're doing because [05:16] it's a great example of how you do [05:18] things right." And so, um, that was you [05:22] all were one of the first, uh, customers [05:24] I actually dug into and learned about as [05:27] kind of a kind of an example for [05:29] everybody else on how to do this stuff. [05:30] So, congratulations on all your success [05:32] so far. [05:34] >> Awesome. Thank you. And then in the [05:36] resources for those joining us, we do [05:38] have a more extensive case study that'll [05:41] give you those numbers if you need or [05:43] something to bring to your own business [05:45] stakeholders to talk about it in that [05:48] way. We're going to get into both the [05:50] business and the tech. But if you need [05:52] just a one cheater, I do recommend [05:54] sharing that. So it's interesting that [05:57] both you Derek and Kelly have worked at [05:59] Speedway for over a decade. So I want to [06:04] talk culture first of where you began [06:06] and how you were able to as you've we've [06:10] already implied at the beginning like [06:12] there were slower releases. It's not [06:14] that there was a lack of psychological [06:16] safety. It sounds like it was more a [06:18] lack of security in the technology of [06:21] making big changes. So, where was your [06:25] culture at and how did you convince [06:27] people to get on board with feature [06:30] flagging and experimentation, [06:33] >> Kelly? I think that's a great one for [06:35] you. [06:37] >> Um, so yeah, I mean, we thinking back a [06:41] couple years now, we do two week long [06:43] sprints. Um, and we still do two week [06:46] long sprints, don't get me wrong there. [06:47] Um, but it was one big release at the [06:50] end of each sprint and we would come in [06:52] at four o'clock in the morning, [06:54] everybody on deck, you know, no not sure [06:57] what's going to happen because we're [06:58] deploying everything that we worked on [07:00] in those two weeks at the same time. Um, [07:03] so you can imagine there's no quick [07:05] feedback loops. If one thing is wrong, [07:08] we have to completely roll back [07:09] everything. It was just very very kind [07:12] of scary. Um, and then we'd send one big [07:15] long email to all of our stakeholders [07:16] saying, "Hey, this is a long list, but [07:19] this is everything that just, you know, [07:21] went out to production today." Um, so I [07:24] mean, that's kind of where we started. [07:25] It's it was really scary. Lots of all [07:27] hands. Um, [07:30] >> Derek, how do you feel from the [07:31] engineering side of that? [07:32] >> That's exactly it. It was a ceremony and [07:35] we had to change our sleep schedule and [07:38] it was a pain in the butt to go and [07:41] >> everybody get together and deploy [07:43] everything. And Kelly, you referenced [07:45] the the bad part of the deployments, [07:47] right? [07:47] >> Outages happen. We have to go back two [07:50] weeks in our memory to figure out this [07:52] one feature caused an issue and we, you [07:54] know, had forgotten about it, moved on [07:56] with different features in our workload. [07:58] Um, but the other thing that we missed [08:00] out on was what worked, [08:02] >> you know. Um there's the stuff that [08:05] broke obviously, but what worked and we [08:07] couldn't double down on that. So we did [08:10] a deployment and maybe things shifted [08:12] around and we saw an increase in AOV, [08:15] increase in revenue, whatever. But if [08:18] somebody came up to us and said, well, [08:20] what feature did that? Everybody could [08:22] claim that their feature did it. Nobody [08:24] really had the proof that this is the [08:27] actual thing that caused the the lift [08:29] and we should continue on that thread. [08:32] So you kind of get it on both sides. [08:34] There's carrots and sticks for you don't [08:36] want to do the bad stuff, but you really [08:37] do want to follow the trail of the good. [08:41] >> Okay. [08:41] >> So, [08:42] >> yeah, go please, Kelly. [08:44] >> I was just going to say, yeah, just to [08:45] kind of finish off on that comment, um, [08:48] we follow like the modern agile [08:49] principles very closely and one of those [08:51] is, you know, to deliver value [08:52] continuously. So, we knew we weren't [08:54] doing that. We knew there was a problem. [08:56] Um, how could we move forward with that? [08:58] And I think that that kind of leads into [09:00] the rest of the conversation. [09:03] >> Yeah. I I think maybe Aaron, you might [09:05] be best define to define what feature [09:08] flagging is. And then I'm really curious [09:10] because there is an a lot of options for [09:14] aggressive delivery. But I'm curious why [09:16] that was chosen as the solution for [09:18] this. So Aaron, if you want to give us a [09:20] quick definition of a feature flag. [09:21] >> Yeah, I can give you I can give you a [09:23] real quick one. I mean, most people are [09:24] are aware. I think at this point, most [09:26] dev teams use feature flags in one form [09:28] or another. They may be very rudimentary [09:31] DIY or they may have a vendor like [09:34] harness that are helping them uh manage [09:36] those and scale those. But basically, [09:39] it's a toggle. You're turning a feature [09:40] on or off. And there's a lot of reasons [09:42] you want to do that, right? The classic [09:44] one is, hey, I want to release this. In [09:46] our case, since we're a vendor, we want [09:48] to be able to turn on, let's say, a beta [09:50] feature for someone. So, we build out [09:51] the feature, put it behind a feature [09:53] flag, and then we can turn that on just [09:55] for a subset of users that are doing the [09:58] the beta test, right? That's how I first [10:01] came in contact with this years and [10:02] years ago, but it's evolved way beyond [10:05] that, right? And and when you add [10:07] experimentation on on that as well, now [10:10] you're able to actually uh do exactly [10:12] what Dererick and Kelly just said, [10:14] right? which is you're not just it does [10:16] solve the problem of these massive [10:18] deployments that you want to break up [10:20] into their uh individual components or [10:22] individual features that you can then [10:24] turn on and off but it's also a way to [10:26] test those and see if they're working. [10:28] So, you turn on a flag, but you only [10:30] turn it on for maybe 10% I think as you [10:33] alluded to earlier on, right, of your [10:36] user base. And maybe that's a maybe [10:38] that's a a progressive roll out or or [10:40] maybe you turn it on just for your uh [10:42] power users, for example, because [10:44] they're going to give you more feedback [10:45] and you're going to know faster whether [10:47] that thing is working. So, there's all [10:49] kinds of various segments you could turn [10:50] on those flags for. And then, of course, [10:52] you experiment on the other end. You set [10:54] up an experiment and say, "Hey, is this [10:56] impacting my checkout rate, right? Or or [10:59] or my cart size or did I order an extra [11:03] burger with my uh fast food order [11:06] because I was testing this new way to [11:08] present this data to you and to present [11:09] an offer to you?" Right? There's all [11:11] sorts of ways to use feature flags. And [11:14] the biggest one we're hearing, of [11:15] course, today is just dealing with the [11:17] velocity of code being generated, right? [11:19] There's so much code that's coming down [11:21] the pike. And um you need a way to be [11:24] able to deal with that. And the last [11:26] thing you want to do is have, you know, [11:29] 200 300% more features that you're [11:31] releasing and still be in that old mode [11:33] of releasing in one big chunk, you know, [11:35] every other week or whatever. Like [11:37] that's just it's just setting yourself [11:39] up for some sort of a risk or some sort [11:41] of sort of a vulnerability to creep in. [11:44] >> Okay. So let's concrete that a bit [11:46] because a feature release could be [11:49] anything from we're going to change the [11:52] entire language of the app or a foreign [11:56] or programming or it could be something [11:59] so small as like a different colored [12:01] button. So can we get into some of the [12:03] experimentation practically y'all did at [12:05] Speedway or maybe your first experiment [12:08] even how you figured out we're going to [12:10] try that first. That's nerve-wracking. I [12:13] remember the most successful experiment [12:17] that I've done and that was we redid our [12:21] product page. We had an old product page [12:24] and like Aaron said, we were able to do [12:26] a progressive roll out on that and this [12:29] one's near and dear to my heart because [12:31] I was leading it. I was I was running [12:34] the project and I could see the results [12:36] of the experiment [12:38] positively influence business outcomes. [12:42] And that was a fairly large experiment [12:44] on the whole, but once it went out, we [12:48] were able to do legitimately what you [12:50] just said, Jennifer, we were able to do [12:51] an experiment on the button color. Like [12:54] there's a reason why Amazon's button [12:56] color is orange. Orange and green have [12:58] been measured to perform the most. Uh [13:02] that's not an accident. They did a test [13:04] on it and they did a test on the button [13:06] color and it ended up successful for [13:08] them. So you can be really nitpicky with [13:12] with your experiments if you want to. [13:15] >> So yeah, explain more about this. So you [13:17] did a rebrand of your product page, I [13:19] guess, and instead of So you still did [13:22] like [13:24] because often, [13:26] >> especially developers get very precious [13:28] about their websites. [13:30] Um, let's call it what it is. And a lot [13:32] of them built their own even though [13:34] that's not their unique value [13:35] proposition building website. uh [13:38] respectfully of course say you got [13:40] precious perhaps but a lot of times it's [13:44] this really spoken to someone in content [13:47] arduous process to get that up there and [13:50] to make that change. So how did you did [13:53] you make it in still like a switch from [13:56] one to the other or did you [13:58] progressively change? So it still was [14:00] like a full rebrand launch but then when [14:02] you launched you had all these toggles [14:04] going. [14:06] >> Yeah. So there was some leadup discovery [14:10] work that we did before when we were on [14:13] the old product page. We were able to go [14:16] add feature flags to it and toggle [14:18] certain features on and off to [14:20] understand what the new pages needed [14:23] like what's really important, what's [14:24] really driving customer behavior. Then [14:26] there was this big toggle where we went [14:29] from the old to the new. We changed the [14:32] tech stack underneath of it. But then [14:34] once we deployed fully that new and [14:38] measured it to have a positive outcome, [14:40] there was the discovery work again where [14:43] we were doing more smaller [14:44] experimentations. We were working on [14:46] stuff like recommendation sections. We [14:48] were working on stuff like what [14:51] placement of images is the most [14:53] effective to to our customers. So [14:56] there's kind of this small big small [14:58] cadence that we use specifically for [15:00] that project. But you can use any like [15:05] you can put anything big or small behind [15:07] a feature flag. Um they're very very [15:09] flexible. [15:11] >> Wow. Okay. [15:13] >> And I would say one of the one of the [15:14] lesser known I would I would just say it [15:16] really does when you start looking at [15:18] how to use feature flags, it really does [15:19] open your mind a little bit. Um you know [15:22] we're talking about basically you know [15:24] testing features but you could also use [15:25] it as part of a migration. I don't think [15:27] that happened in this case, but you [15:29] could certainly use feature flags as a [15:31] way to move from on-prem to cloud or [15:34] from one database to another database. [15:36] uh one version of your application [15:38] that's built on a particular uh language [15:41] and structure, you know, [15:44] um maybe you're using uh moving into [15:47] containers and Kubernetes from more of a [15:49] static inst there's just like all kinds [15:51] of things once you start getting into [15:53] this world of oh what can I control [15:56] easily with this thing um that your mind [15:59] starts kind of expanding the [16:01] possibilities [16:03] >> and honestly in the world of AI I and [16:06] the cost of cloud etc. There's also [16:09] people experimenting with moving [16:11] workloads back they may be cloudnative [16:13] workloads but back to onrem and figuring [16:16] out which is more efficient more [16:18] effective for a different workload. I'm [16:22] very curious about this process. So did [16:25] you have to hire people that were [16:29] feature flag [16:31] speakers or proficient? Did you expand [16:34] your team because you had you were [16:37] running these experiments? How did you [16:39] perform training, onboarding [16:41] into the feature management? [16:43] >> We did not have to hire anybody. Our [16:45] team picked it up. It was very [16:47] intuitive. [16:50] Yeah, coming from someone who does not [16:52] have a technology background actually um [16:55] I it's was very quick for me to pick up [16:59] um figure out you know exactly I care [17:01] more about the metrics of something [17:02] exactly where to go how to look at those [17:04] things um so even from like a [17:06] non-technological [17:08] aspect I would say it was just super [17:09] quick to pick up [17:11] >> and Jennifer just to say one other thing [17:13] this kind of doubles down on something [17:15] that you said too Aaron the feature [17:17] flags themselves don't necessarily need [17:19] to be experiments every time. There's [17:23] such a thing as a safety feature flag. [17:25] And I think that for developers, like [17:29] having safe code is is incredibly [17:32] important. So if a developer can see, [17:35] oh, I can wrap this new feature that I'm [17:37] building that I'm not going to have any [17:40] test on. I have no intention on testing [17:42] it, but it ensures the safety of the [17:45] overall experience. and if something's [17:47] broken on a deployment, I can just go [17:49] turn it off immediately. I think that [17:51] was a really good selling point for the [17:53] engineering team, too. And it also [17:55] helped the the adoption of it. I think [17:58] maybe the more business background, the [18:00] business-minded people really like the [18:02] experiments and like the business [18:04] outcome side of it. Like that that's [18:06] makes intuitive sense to that crowd. But [18:08] the safety and honestly reputation that [18:12] comes with engineers is my code works [18:16] and and that helps the adoption of [18:18] feature flags from the from the [18:20] technical side. [18:21] >> That also closes that important feedback [18:24] loop that's so that cognitive load that [18:26] developers are holding and at a two week [18:30] sprint you maybe didn't even remember [18:32] what you did [18:34] >> to then be able to fix it. But if you [18:36] have an immediate feedback loop that [18:37] closes that loop and it's also an [18:39] adrenaline rush to be accepted or maybe [18:42] not and have to fix and scramble it, but [18:44] at least it comes back and you can fix [18:46] it. [18:48] >> Yeah. Certainly a lot better than [18:49] rolling back a huge deployment with [18:52] tens, 20s, however many features you [18:55] deploy, right? Because that's the [18:57] alternative. That's why it's kind of the [18:58] safety blanket, if you will, on a [19:00] feature-byfeature basis. Otherwise, you [19:03] know, who knows? Is that roll back going [19:04] to succeed? You know, all those features [19:06] that you rolled out, whoever was exposed [19:08] to those now doesn't have those anymore. [19:10] Where if you're dealing with feature [19:12] flags, it's you're dealing with it on a [19:13] very, very tight window, very tight [19:17] application where you can just say, "Oh, [19:18] I'm just going to take that one thing [19:20] that's failing and I'm going to turn it [19:21] off. And by the way, I can give that [19:23] functionality to my incident team." So [19:26] they they're enabled to do it. I don't [19:28] know if if at Speedway how you deal with [19:29] that, but I mean it it does kind of [19:31] democratize the ability to to handle [19:34] when things go bad, so to speak. [19:36] >> Yeah. And and one other thing that we [19:38] can do just to double down on that point [19:40] is you mentioned targeting before, [19:42] Aaron. We can target ourselves first. We [19:45] can do a production deployment and [19:47] target, you know, our team, you know, [19:50] and stakeholders directly and we can [19:54] test it out on oursel in the production [19:56] environment. I mean there's a joke about [19:58] do you test in production? We do. We do [20:00] test in production. We just don't test [20:02] it on everybody. We we can specifically [20:04] target ourselves and ensure that it's [20:07] going to be good so no customers are [20:09] ever going to have that poor experience [20:11] if that were to happen. [20:13] >> Do the business stakeholders or other [20:15] stakeholders ever become a bottleneck in [20:17] that situation? because then they're [20:19] like, "Well, use it." And then they're [20:21] like, "Well, [20:23] you could change it a bit." [20:27] >> Yeah, I'm sure that that does happen. [20:30] You know, I don't think you're ever [20:31] going to automate that away. Um, [20:34] >> yeah, [20:35] >> one thing it does allow us to do is if [20:38] if that conversation does come up and a [20:41] business stakeholder is like, "Hey, I'm [20:42] not so sure about this." You can release [20:44] to a segment of your customers. you can [20:47] release to 5 10%. And that's when a test [20:50] helps out a ton because if it's just my [20:54] word against your word, that's that's [20:56] one thing. But if we actually have data [20:58] to back it up, that can kind of drive [21:00] the conversation in a less political, [21:02] more advantageous way. [21:06] >> How does it work on the other side? Have [21:08] you had, and of course nothing giving [21:09] away the secret sauce, but have you had [21:12] features that just didn't work? Like [21:16] customers responded differently, [21:18] stakeholders responded differently, and [21:22] you just were surprised. How do you deal [21:24] with that then? [21:28] I think uh at that point, you know, it [21:30] depends because for example, um we [21:34] recently went to live with a fully [21:36] landed cost shipping option for our [21:38] international customers. Um and what we [21:40] did is we started out about 5% just to [21:43] make sure that you know we could see [21:45] some orders get through full conception [21:48] from uh order place to ship to getting [21:51] through customs, make sure everything, [21:53] you know, follows that path correctly. [21:55] Um, and we actually found at about [21:58] actually we went to 100%. So we got all [22:01] the way to 100% found out that there was [22:03] an issue once the packages were getting [22:05] to customs. [22:06] >> Um, they were being charged again by [22:09] UPS. So what we were able to do is we [22:12] turned that feature flag back off. Um, [22:14] realized, okay, something's going on [22:16] here. It gave us that time to be like, [22:18] okay, we're stable. Obviously, we have [22:20] it turned off so we know that um we're [22:22] not offering fully landed cost anymore. [22:24] Um so the expectation is correct on the [22:27] customer's side of things, but then it [22:29] gave us that moment to be like, okay, [22:30] we're stable now. We can figure out [22:32] what's going on, where the problem is. [22:34] Is it on our end? Is it on UPS's end? Um [22:37] so that was something that we were able [22:39] to, you know, really quickly kind of [22:41] dive into. Um and you know, this isn't [22:45] the only one that we found out, oh no, [22:47] we have to turn it off. you know, those [22:48] things happen. Um, but it just really [22:50] gives you that quick run back to be [22:52] stable back to where you know things are [22:55] okay and then give you that time to dive [22:57] into what's going on. [22:59] >> Yeah. And it's really easy to get [23:00] pessimistic about experiments that [23:02] didn't pan out because you're excited [23:04] about them. [23:05] >> But I think it's a win to learn. It's a [23:08] good thing that you know not to do that. [23:11] Um, learning in either direction, [23:13] positive or negative, is a good thing. I [23:16] would say maybe one of the harder points [23:20] is if you do an experiment and you have [23:23] inconclusive results. [23:26] That's kind of a tough one. You have to [23:27] choose a direction to go. Um but I'd [23:31] rather run an experiment than not run [23:33] one. [23:34] >> Okay. Another use case brought from the [23:37] audience is so thank you. Uh you can [23:41] have feature flags for fallback for [23:43] maintenance states to show a message. [23:46] Also for areas of your code that you [23:48] have a dependency on a third party such [23:50] as figuring out state taxes to apply so [23:53] you handle outages you can't control. I [23:57] think that's great too because I think [23:59] both examples that we just talked about [24:02] are things are not as much as you want [24:05] to experiment things are not always in [24:07] your control. they may be in the world's [24:10] largest shipping company's control that [24:13] you are a very valued partner. So you [24:16] have to figure that out and I think [24:19] that's a really valid point. I'd love to [24:21] get into the practicality now. Derek, do [24:23] you want to tell us about how you [24:25] actually go about what a feature flag [24:27] looks like from the technical end? It [24:29] sounds like Kelly on a more my style [24:33] view. Yes. [24:34] >> That it's business and that's important [24:37] too. We know with platform engineering [24:38] we need that single pane of glass so [24:40] business stakeholders can see but I'm [24:42] sure we have some technical listeners. [24:44] So do you want to go into that how that [24:46] works Derek? [24:47] >> Yeah so how we implemented it was we [24:50] have a a shared SDK library uh in our [24:55] code in our service harness [24:58] specifically. [25:00] Um what that allows us to do is that [25:04] shared code is more of a utility and we [25:07] can call that from anywhere in our [25:09] codebase. So that means you have this [25:11] very thin call, this thin layer um that [25:16] surrounds your feature or your third [25:18] party. I was going to actually double [25:20] down on that. We have a a feature flag [25:22] that controls our warehouse right now [25:24] when we have inclementware weather. If [25:27] we have poor weather and we can't ship [25:28] out, we can turn our warehouse off with [25:30] a feature flag. [25:32] >> Wow. [25:33] >> Um, yeah. So, so you can have this be [25:37] loadbearing in your overall operations. [25:40] Uh, but yeah, it makes it very very [25:43] easy. It's a thin call that you would do [25:46] and then you just surround your code in [25:48] an if statement. If this feature is on, [25:51] do this. Else do the old thing. [25:56] Okay. You had an image you wanted to [25:58] share, right? Do you want to explain [26:00] that? Bring that up. Like a good thing. [26:02] I'm still amazed by that example, by the [26:05] way. [26:05] >> Yeah. Just to double down on that, like [26:08] there's um permanent feature flags that [26:11] we have. So, I kind of have this example [26:13] that I shared with with Jennifer before [26:14] this. Um this is an example of a really [26:17] complicated permanent feature flag that [26:20] we have. Um [26:23] we have a very complicated [26:25] recommendation solution that we have [26:28] measured to be positive like in terms of [26:31] business outcome. This has increased our [26:32] revenue quite a bit. Uh what we've done [26:36] is we've created just a JSON object that [26:38] we've put inside of our feature flag [26:41] configuration [26:43] and it allows us to test very [26:46] complicated [26:47] uh examples against against [26:51] uh just head-to-head. If you are blessed [26:53] to have just copious amounts of traffic, [26:56] you can actually run more than just AB. [26:59] You can do an ABCD test if you want to [27:02] and 25% of traffic goes to each and you [27:05] can learn pretty rapidly. Um so this is [27:08] an example here. It kind of does two [27:11] things. We're able to do the experiment [27:13] on head-to-head. Is it better to put [27:15] mandatory Rex in front of consumable Rex [27:17] or vice versa? That could be a [27:19] hypothesis that we would test out. But [27:22] say there's just a a simple text change [27:24] that we want to make to our title. Uh we [27:27] can just go in there and change the [27:29] title without ever having to do a [27:31] deployment. Uh it's instantly realized [27:33] on the client. [27:36] >> Wow. Fantastic. [27:38] >> Yeah, I think that's an important um as [27:41] people are learning how to do this or [27:42] maybe trying to create their own DIY [27:45] feature flag uh practice. Um, it it is [27:49] kind of amazing because all this runs [27:50] client side, how quickly you can change [27:52] things. You change a feature flag and [27:55] it's I I I can't say it's instantaneous [27:57] from a mathematical perspective because [27:58] that's not true. But when you're [28:00] comparing it to to rolling out a new [28:02] version of your app, let's say on your [28:04] phone, hoping someone downloads it or [28:06] whatever to get that upgrade. being able [28:08] to just flip flip a toggle like [28:10] literally have it appear within moments [28:14] uh for your customer and then be able to [28:16] see results come in almost immediately. [28:20] It really is gamechanging. [28:23] >> Well, thank you. I'm still obsessed with [28:25] that idea of the weather shutting down [28:27] the warehouse. That is the world we're [28:30] entering [28:32] >> for sure. [28:33] >> Yeah. And it's super convenient. Like it [28:35] used to be a panic, right? like we [28:37] would, oh no, we have to go and do a [28:39] deployment to turn off our warehouse and [28:42] you want me to do that right now at 6:00 [28:44] in the morning because it's snowing [28:46] outside. Now it's just a feature flag. [28:48] We did that that work once and we have [28:50] the ability to toggle it whenever we [28:52] want to. [28:53] >> And you would call that like an [28:55] evergreen feature flag. We had talked [28:56] about feature flags that you reuse or [28:59] keep running, but feature flag debt is [29:02] like everything else. Everybody's got [29:03] their zombie servers running. Everyone's [29:06] got their zombie microservices running. [29:07] Like how do you combat feature flag [29:11] debt? How do you decide what is a [29:13] permanent feature flag versus an [29:14] experiment? When do you turn these off? [29:16] >> Yeah, that's a great question. I would [29:18] say our evergreen feature flags are ones [29:21] that wrap third parties. Uh like the [29:24] example given in chat, the ones that you [29:26] know, our warehouse, that's an evergreen [29:29] one. Our recommendations are going to be [29:30] evergreen. Um, we've we've also got a [29:34] evergreen feature flag that targets [29:36] internal users. So, if you are a [29:40] Speedway Motors employee, you can see [29:42] certain uh data points on our site that [29:46] other people won't. Mostly it's like [29:48] search investigation. [29:50] Um, so there's a couple dedicated ones [29:53] that we have. The debt does pile up. Um, [29:57] the best way to do like I would say the [29:59] non- evergreen ones, the ones where you [30:01] had an experiment and then you uh [30:04] concluded the experiment, then you moved [30:06] on with your life or you just had that [30:08] safety feature flag where your code was [30:10] behind a feature flag, you were able to [30:13] deploy it, test it on yourself, not an [30:16] experiment, and then just turn it on. [30:18] Those feature flags do hang around. They [30:21] do hang around. One thing that we have [30:23] done to combat that is uh harness has an [30:27] API and we have a task that will go and [30:31] query harnesses API to see all of the [30:33] feature flags that are outstanding and [30:36] if it's not a dedicated evergreen [30:39] feature flag if we haven't set that up [30:41] as a as a tag for for permanent uh it [30:45] will alert us. It'll say this thing's [30:47] been on for over a month. Are you sure [30:49] you want to keep it? And what is the [30:51] risk of leaving feature flags on? [30:55] >> Very low. Very low. I guess you could [30:58] accidentally go back and turn it into [31:01] the incorrect state and then you built [31:03] code around it where now your feature [31:06] flag being in the off state would not [31:09] act nicely with the new code that you've [31:11] written. Um [31:14] >> that's just good code hygiene. Yeah. [31:16] >> Yeah. [31:18] the the feedback loop on that would be [31:20] nice if we can go in and remove that [31:22] from code more [31:25] uh eagerly. Maybe that's an application [31:27] for AI. [31:29] >> Okay. [31:29] >> Yeah. I I can't go into the details on [31:31] that, Derek, on this one, but um [31:34] >> uh stay all I can say is stay tuned for [31:37] for just a little bit longer and uh [31:39] we'll we'll we'll have a follow-up and [31:41] see how that's going, I guess. [31:42] >> But there are other mechanisms as well. [31:45] I mean uh what Dererick is saying is is [31:47] really popular actually because we do [31:49] have a concept of pipelines and policies [31:51] in place in the product. So you can set [31:54] a step in your in your uh pipeline for [31:57] example that will say uh go you know [32:01] after this has hit 100% after this flag [32:03] has been rolled out to 100% you have a [32:05] step that's automated. it's part of your [32:07] process that says go clean up this code, [32:09] here's where it lives, and then make [32:11] sure that gets deployed and updated as [32:13] part of your next um deployment. So you [32:16] absolutely can can at least automate it, [32:18] make sure you have policies in place to [32:20] enforce it so that people do go in and [32:24] um clean that stuff up because you can [32:25] imagine, you know, we're talking about [32:27] individual use cases here, but those add [32:29] up, especially for companies that have [32:33] thousands and thousands of developers. [32:35] Um that the amount of code that's that's [32:37] kind of left hanging out there, it's [32:39] important to go clean it up at the end [32:40] of the day. [32:41] >> And it's just going to be more code with [32:43] AI. We're just going to have more of [32:45] everything. It's going to amplify all [32:46] the bad habits. So, let's get good [32:49] hygiene and habits in place and avoid [32:51] more debt. [32:53] >> 100%. [32:54] >> Love this audience question, so I pick [32:56] it eagerly. How do you measure the [32:58] impact of the feature that's toggled on? [33:01] What metrics do you use? Do you pair [33:02] with AB testing, etc.? Kelly, I think [33:06] business probably super fun [33:08] functionality, I guess, supersedes [33:09] business, but business probably [33:11] supersedes [33:13] everything else in this, right? [33:16] >> Yeah. I mean, there are metrics built [33:17] into the harness app. So, um, you know, [33:20] I think Erin touched on it just a little [33:22] bit earlier, but um, as soon as you turn [33:24] something on, it is immediately starting [33:26] to read that data. um and it is [33:29] comparing you know the certain [33:30] percentage of what you have enabled or [33:33] the certain segment of group of people [33:34] that you have enabled to you know the [33:37] off um control group right so um you [33:40] know we try to run our experiments for [33:42] at least two weeks just to have a good [33:45] amount of data um but the metrics are [33:48] definitely built right there in the [33:49] harness app um and I think we can um you [33:53] know really narrow in on the ones that [33:54] we value the most um and look at them [33:57] from that perspective effective. [33:59] >> Yeah. And to follow that up, there's a [34:01] lot of metrics that we have created [34:02] specifically. [34:04] >> There are very specific use cases like [34:07] we have user added a year make model to [34:10] their garage. That's only going to be [34:12] applicable for an automotive retailer. [34:14] But on top of that, you have your [34:16] general e-commerce ones. Revenue, AOV, [34:20] conversion rate. We're watching those [34:21] like a hawk. Um [34:25] the closer to the metal that you can get [34:27] the better. So if you are like changing [34:32] your search algorithm, you should test [34:35] for search product engagement. You [34:38] shouldn't look at the overall revenue [34:40] increase. Although it is important to [34:42] keep an eye on that. It's just a couple [34:44] steps removed from the feature that you [34:47] are changing. Uh I would encourage I [34:50] guess to have both. There's no reason to [34:52] not have both as key metrics. Uh but [34:55] definitely look as close as you can to [34:57] the feature that you are changing. [35:00] >> Absolutely. [35:02] >> And AOV for those like myself not in [35:05] e-commerce stands for average order [35:08] value says the Google AI results. So I [35:11] hope that's [35:11] >> that is correct. I'm sorry. [35:13] >> No, it's fine. I was like should I know [35:16] this? Should I I don't work in [35:17] e-commerce. It's okay. I don't know this [35:18] to say online. Uh, [35:20] >> threeletter acronyms. [35:22] >> So many threeletter acronyms. [35:23] >> Business and tech together. It's worse [35:25] than the UN. We have another great [35:28] audience question I want to get to [35:30] before we wind down. How do you manage [35:32] testing when you don't really know which [35:34] other flags will be enabled when you [35:36] release? Oh, and I guess this comes when [35:39] we talk about the scale too of a bank, [35:42] which is the largest tech companies for [35:44] a lot of people. [35:46] How I'm understanding that question is [35:50] the test that you're writing like the [35:52] code feature has multiple uh feature [35:57] flags enabling it or maybe there's like [35:59] a nesting there's some sort of [36:01] dependency between the two. Um on that I [36:05] would say communication is your friend. [36:07] Make sure that you're talking to the [36:08] person that is also in the area that [36:11] you're working on. Um I would say the [36:15] very best way to do it just from a [36:18] statistical standpoint would be to have [36:22] every condition covered. So make a truth [36:25] table off off on on off on on um and [36:30] figure out which one wins in those. Now [36:32] it could be the case that some of those [36:34] are not viable where there the [36:36] dependency is such that [36:38] it would cause an error state if one's [36:40] on and the other's off. Um, if you can [36:44] anticipate that, make sure it doesn't [36:45] happen in code. If if that error case is [36:49] met, uh, just do the default. Just do [36:52] what was happening before and then alert [36:55] on it so that the team can can go in and [36:58] and fix that issue. Okay, that's [37:00] definitely a case for avoiding your [37:03] future flag debt going up too much and [37:06] you know keep good hygiene as well so [37:08] that doesn't happen because I honestly [37:10] hadn't thought of that but it's dolls [37:14] nesting dolls all the way down sometimes [37:16] or turtles all the way down. So [37:18] >> yeah. [37:18] >> Yeah. Go. [37:20] >> Oh, I was just going to say like we have [37:22] a smaller team so it hasn't happened all [37:24] that much. we're kind of in a whole [37:25] bunch of different areas or one team's [37:27] over here and another team's over over [37:29] there. So, it doesn't happen all that [37:31] much in practice as far as I know. Um, [37:34] but for a larger team, I could [37:35] definitely see that happening. [37:37] >> Aaron, have you heard anything at some [37:39] of the enterprises? Um I mean it is [37:42] something you have to watch out for. But [37:44] I think the if you look for um uh best [37:49] practices for example or maturity of [37:52] companies that are doing this well they [37:53] try not to let that happen. Right? You [37:56] want to be testing something very very [37:57] specific and then you want to be doing [38:00] quick experiments on whether that moves [38:03] the needle or not and then moving on [38:05] from there uh when you can. So you [38:07] definitely don't want to be combining or [38:09] even in your code having multiple [38:12] triggers if you can help it because it [38:13] just makes it more difficult obviously I [38:16] guess to to figure out what what is [38:18] actually going on. So the more specific [38:20] you can be um and deliberate you can be [38:22] and then the more experiments you can [38:24] run um is going to lead to better [38:27] results down the road. [38:29] >> Yes. And and I will say just to add on [38:31] uh Derek, correct me if I'm wrong at any [38:33] point here, we also have a connection uh [38:35] with harness in one of our Slack [38:37] channels. Um so if anyone is changing a [38:40] future flag or an experiment um it will [38:42] automatically send a message to that [38:44] Slack channel with the environment what [38:46] was changed um etc. So all of our [38:49] developers have access to that. Now [38:51] that's on you know the engineers to be [38:53] reading that but it is just another way [38:56] to like hey this is changing. Um, so [38:59] just that that there's more [39:00] communication out there as well. [39:02] >> Yeah. And and of course, like I said [39:04] before, if you're if you're using [39:06] policies and pipelines in the product, [39:08] you can also set that as a check right [39:10] before it goes live. So you can c you [39:13] can catch things and you can do hooks [39:15] into Slack and all kinds of other [39:17] applications as well. So yeah, [39:20] >> and when it comes down to it, we're [39:22] talking about technology, but people and [39:24] processes are what tend to be the most [39:27] important part of any technical change. [39:30] So as we wind down and since stating [39:33] goals make them more likely to come [39:35] true, where do you hope that your [39:38] organization goes next on your future [39:40] flag journey, how you're going to [39:41] leverage them next? [39:47] That's that's a good question. [39:50] >> Is AI going to be involved or no? [39:54] >> Most likely, yes. Um, we are very [39:56] heavily experimenting with AI at this [39:58] point. So, you know, we're always [39:59] looking for ways to evolve. I would say [40:01] we're very much proactive in many [40:04] different aspects of looking into, you [40:06] know, how can we always be better? Um, [40:08] AI is definitely on that. Uh, Derek, do [40:10] you have any specifics there maybe? [40:13] >> Yeah. Uh, one annoying experience that [40:16] I've had with AI is like the [40:18] configuration on the requests to an AI [40:21] agent. Uh, you can change from chat GPT [40:26] to the newest version that Anthropic has [40:28] out and they've been updating versions [40:31] fairly often. One convenient thing that [40:34] I've done is I've made a feature flag, a [40:37] permanent one that just has what model I [40:39] want to use in the prompt text so that I [40:42] don't have to do a deployment every time [40:44] I change my integration with AI. I can [40:47] just go and change the model in the [40:49] configuration. [40:51] >> Well, within the feature flag, you then [40:53] update that notification. So it's on [40:56] your list like hey when this update [40:58] happens every other week probably right [41:01] now at the speed we're going at you set [41:03] it it's like on your list of things to [41:06] do when you're of your own hygiene your [41:08] own best practices with AI. [41:12] >> Uh if there's not a problem I won't go [41:14] and change it. But when I'm finding that [41:18] one certain model is not doing what I [41:20] want it to do and there's another one [41:22] out there, maybe it maybe I'm changing [41:24] from like a a thinking model that has [41:27] the ability to scrape websites to just a [41:29] nano or something for cost reasons. I [41:32] can I can do that from a feature flag [41:35] via that configuration. Very similar to [41:38] the recommendation example I showed [41:39] earlier. Um it makes it a lot easier so [41:43] that I don't have to go and deploy all [41:44] my services to up update my codes [41:48] integration with a model. [41:50] >> Yeah. And this goes down a pretty big [41:51] rabbit hole of token use and all kinds [41:55] of other things, right? Where you could [41:58] very easily switch processes from one [42:00] model to another, from one prompt to [42:02] another um using feature flags um in a [42:06] way that you know could could actually [42:08] impact the bottom line. It could make [42:10] your results better or worse. I mean, it [42:12] has all sorts of implications to how [42:14] you're using AI in your in your [42:16] environment right now. That's a great [42:18] example. [42:19] >> So important. I wish you luck in your [42:22] fle feature flag progressive delivery [42:25] experimentation. Somebody had to make it [42:27] illiterative to make me more challenged [42:29] as a host. Uh, and I'm so grateful to [42:32] the audience. First of all, you've [42:34] brought great questions. There is a free [42:37] harness demo that you can get in the [42:39] resources so you can play around with [42:41] feature flags because experimentation [42:43] starts with experimentation [42:46] and we know developers especially want [42:48] to get their hands on it. But I love [42:50] that also the business stakeholders can [42:52] too. And thank you so much to Derek, [42:54] Kelly and Aaron and have a great day [42:57] all. Thank you. [42:59] >> Yeah, thank you very much. It's been [43:00] great. [43:00] >> Thank you. [43:03] Hey, hey, hey.  📄 **Full text:** ![[Full Document Contents/Articles/Case Study Feature Flags IRL.md|Case Study: Feature Flags IRL by YouTube]] ## Highlights - came in contact with this years and years ago, but it's evolved way beyond that, right? And and when you add experimentation on on that as well, now you're able to actually uh do exactly what Dererick and Kelly just said, right? which is you're not just it does solve the problem of these massive deployments that you want to break up into their uh individual components or individual features that you can then turn on and off but it's also a way to test those and see if they're working. So, you turn on a flag, but you only ([View Highlight](https://read.readwise.io/read/01kqzg0k1t9kx2154skthhxs5j)) - Tags: 👻 ai highlighted - important. So if a developer can see, oh, I can wrap this new feature that I'm building that I'm not going to have any test on. I have no intention on testing it, but it ensures the safety of the overall experience. and if something's broken on a deployment, I can just go turn it off immediately. I think that was a really good selling point for the engineering team, too. And it also helped the the adoption of it. I think maybe the more business background, the ([View Highlight](https://read.readwise.io/read/01kqzg0k4pnekgja3kx1wvjvn4)) - Tags: 👻 ai highlighted - business-minded people really like the experiments and like the business outcome side of it. Like that that's makes intuitive sense to that crowd. But the safety and honestly reputation that comes with engineers is my code works and and that helps the adoption of feature flags from the from the technical side. >> That also closes that important feedback loop that's so that cognitive load that developers are holding and at a two week sprint you maybe didn't even remember ([View Highlight](https://read.readwise.io/read/01kqzg0k6m3sjnnq1g4yeday9v)) - Tags: 👻 ai highlighted - are things are not as much as you want to experiment things are not always in your control. they may be in the world's largest shipping company's control that you are a very valued partner. So you have to figure that out and I think that's a really valid point. I'd love to get into the practicality now. Derek, do you want to tell us about how you actually go about what a feature flag looks like from the technical end? It sounds like Kelly on a more my style ([View Highlight](https://read.readwise.io/read/01kqzg0kbjsj6ng6t9aann7pkc)) - Tags: 👻 ai highlighted - deployment. Uh it's instantly realized on the client. >> Wow. Fantastic. >> Yeah, I think that's an important um as people are learning how to do this or maybe trying to create their own DIY feature flag uh practice. Um, it it is kind of amazing because all this runs client side, how quickly you can change things. You change a feature flag and it's I I I can't say it's instantaneous from a mathematical perspective because that's not true. But when you're ([View Highlight](https://read.readwise.io/read/01kqzg0kfp470exdc8jptepv3r)) - Tags: 👻 ai highlighted - entering >> for sure. >> Yeah. And it's super convenient. Like it used to be a panic, right? like we would, oh no, we have to go and do a deployment to turn off our warehouse and you want me to do that right now at 6:00 in the morning because it's snowing outside. Now it's just a feature flag. We did that that work once and we have the ability to toggle it whenever we want to. >> And you would call that like an evergreen feature flag. We had talked about feature flags that you reuse or keep running, but feature flag debt is ([View Highlight](https://read.readwise.io/read/01kqzg0kq3kzppeg5ns3wg0fg8)) - Tags: 👻 ai highlighted - had an experiment and then you uh concluded the experiment, then you moved on with your life or you just had that safety feature flag where your code was behind a feature flag, you were able to deploy it, test it on yourself, not an experiment, and then just turn it on. Those feature flags do hang around. They do hang around. One thing that we have done to combat that is uh harness has an API and we have a task that will go and ([View Highlight](https://read.readwise.io/read/01kqzg0kw8t10bmgenhz2pb2fg)) - Tags: 👻 ai highlighted - the audience. First of all, you've brought great questions. There is a free harness demo that you can get in the resources so you can play around with feature flags because experimentation starts with experimentation and we know developers especially want to get their hands on it. But I love that also the business stakeholders can too. And thank you so much to Derek, Kelly and Aaron and have a great day all. Thank you. >> Yeah, thank you very much. It's been ([View Highlight](https://read.readwise.io/read/01kqzg0m6egcmfe9yrm2ecjcag)) - Tags: 👻 ai highlighted