Case Study: Feature Flags IRL  🔗 **Source:** [Case Study: Feature Flags IRL](https://www.youtube.com/watch?v=RAUscXUlZmI) Hello everyone and a huge thank you for joining us today for today's panel. This is a lead dev online engineering and action discussion created in partnership with harness. Engineering action is still a panel but it's kind of a case study is probably the best definition of it of how things are done in the wild. So this discussion will last roughly 45 minutes after which both myself and our esteemed panelists will head over to the lead dev slack to answer some of your questions in the panel Q&A and that channel is called panel Q&A. And for those that are joining us live, you have in the resources and I'll drop this link in the chat. So you have an invite to the lead dev slack. It's a really great community of engineering leaders supporting and learning from each other. So if you give us questions live, we may answer them live. we may run out of time and continue in the Q&A feature uh the in Slack or you can just use the Q&A feature now in Zoom and we will get to your questions either asynchronously or right now. So I do welcome everyone that's joining us live. That's a benefit to joining live but you are deeply welcome if you're joining us later. So yes, Speedway Motors is a tech enabled e-commerce and retail store processing thousands of transactions a day. So that means that every step in their endto-end process from showing up at an ad for search results, someone entering a store in person or a partner shop all the way down to a sale, shopping carts, shipping, everything really matters. But they were dealing with fortnightly releases and you kind of live on edge as a developer like is every change a risk and how do we reduce that risk? So they needed to do something to reduce that blast radius, make it safe to experiment, what we call progressive delivery, which is an umbrella term for reducing that blast radius and releasing as it says progressively to maybe a few of their customers or maybe choosing c chosen customers, specific ones or potential customers, etc. And now with feature management and feature flagging, which is a specific niche we're talking about within the progressive delivery life cycle, speedway engineers and business stakeholders can release a few times a day. And if they want only the first 1% of their traffic. So today we're going to get both sides of that business tech coin with delivery manager Kelly and technical engineer Derek. And as a bonus questioner and feature flagging fan, we're also joined by Harness's Aaron. We're going to ground this conversation in real features tested, how these experiments are set up and managed, ephemeral and evergreen future flags, and more. So, let's get started with introductions. You've probably seen me before here. I'm Jennifer Higgins, and I'm a freelance technology journalist, including with lead dev. And today I'll be joined, as we said, by Kelly, Derek, and Aaron who are ready to share their experience. Can you please introduce yourselves? So, Kelly, do you want to kick off? >> Yeah, absolutely. Hi. Uh, I'm Kelly Bear. I'm the senior agile delivery manager here at Speedway Motors. I have been with Speedway for just over 11 years now, so have seen a lot of evolution on the tech side of things. Um, super excited to talk through things today because of that. um just to pass along how much we have learned along the way and I'm excited to share that knowledge. >> Great, Derek. >> Thank you, Kelly. Uh my name is Derek Stapleton. Like Kelly, I also have a decade of experience at Speedway Motors. Uh my team is primarily responsible for customer acquisition and product discovery as kind of our niche, but I was on the ground floor with the adoption of our feature flagging and the culture shift that went along with it. So I'm very excited to evangelize this stuff because it's changed my life. I love it. We can get into that a little bit more later, but uh Erin, I think you might be next. >> Yeah, I'll go next. So, my name is Aaron Nukem. I'm a senior manager. I work with our uh product called FME. It's not uh something that a lot of people are aware of, but it stands for feature management and experimentation. Uh we acquired Split.io. So, if you are familiar with Split or have used it in the past, that's now part of harness and it uh integrates with all of our other products. So, um yeah, a little bit of a little bit of a transition there for Split, but we're glad to have them. And I will say I've been here for about 6 months and the first thing I heard was, "Hey, have you heard what Speedway Motors is doing? You need to go take a look and see what they're doing because it's a great example of how you do things right." And so, um, that was you all were one of the first, uh, customers I actually dug into and learned about as kind of a kind of an example for everybody else on how to do this stuff. So, congratulations on all your success so far. >> Awesome. Thank you. And then in the resources for those joining us, we do have a more extensive case study that'll give you those numbers if you need or something to bring to your own business stakeholders to talk about it in that way. We're going to get into both the business and the tech. But if you need just a one cheater, I do recommend sharing that. So it's interesting that both you Derek and Kelly have worked at Speedway for over a decade. So I want to talk culture first of where you began and how you were able to as you've we've already implied at the beginning like there were slower releases. It's not that there was a lack of psychological safety. It sounds like it was more a lack of security in the technology of making big changes. So, where was your culture at and how did you convince people to get on board with feature flagging and experimentation, >> Kelly? I think that's a great one for you. >> Um, so yeah, I mean, we thinking back a couple years now, we do two week long sprints. Um, and we still do two week long sprints, don't get me wrong there. Um, but it was one big release at the end of each sprint and we would come in at four o'clock in the morning, everybody on deck, you know, no not sure what's going to happen because we're deploying everything that we worked on in those two weeks at the same time. Um, so you can imagine there's no quick feedback loops. If one thing is wrong, we have to completely roll back everything. It was just very very kind of scary. Um, and then we'd send one big long email to all of our stakeholders saying, "Hey, this is a long list, but this is everything that just, you know, went out to production today." Um, so I mean, that's kind of where we started. It's it was really scary. Lots of all hands. Um, >> Derek, how do you feel from the engineering side of that? >> That's exactly it. It was a ceremony and we had to change our sleep schedule and it was a pain in the butt to go and >> everybody get together and deploy everything. And Kelly, you referenced the the bad part of the deployments, right? >> Outages happen. We have to go back two weeks in our memory to figure out this one feature caused an issue and we, you know, had forgotten about it, moved on with different features in our workload. Um, but the other thing that we missed out on was what worked, >> you know. Um there's the stuff that broke obviously, but what worked and we couldn't double down on that. So we did a deployment and maybe things shifted around and we saw an increase in AOV, increase in revenue, whatever. But if somebody came up to us and said, well, what feature did that? Everybody could claim that their feature did it. Nobody really had the proof that this is the actual thing that caused the the lift and we should continue on that thread. So you kind of get it on both sides. There's carrots and sticks for you don't want to do the bad stuff, but you really do want to follow the trail of the good. >> Okay. >> So, >> yeah, go please, Kelly. >> I was just going to say, yeah, just to kind of finish off on that comment, um, we follow like the modern agile principles very closely and one of those is, you know, to deliver value continuously. So, we knew we weren't doing that. We knew there was a problem. Um, how could we move forward with that? And I think that that kind of leads into the rest of the conversation. >> Yeah. I I think maybe Aaron, you might be best define to define what feature flagging is. And then I'm really curious because there is an a lot of options for aggressive delivery. But I'm curious why that was chosen as the solution for this. So Aaron, if you want to give us a quick definition of a feature flag. >> Yeah, I can give you I can give you a real quick one. I mean, most people are are aware. I think at this point, most dev teams use feature flags in one form or another. They may be very rudimentary DIY or they may have a vendor like harness that are helping them uh manage those and scale those. But basically, it's a toggle. You're turning a feature on or off. And there's a lot of reasons you want to do that, right? The classic one is, hey, I want to release this. In our case, since we're a vendor, we want to be able to turn on, let's say, a beta feature for someone. So, we build out the feature, put it behind a feature flag, and then we can turn that on just for a subset of users that are doing the the beta test, right? That's how I first came in contact with this years and years ago, but it's evolved way beyond that, right? And and when you add experimentation on on that as well, now you're able to actually uh do exactly what Dererick and Kelly just said, right? which is you're not just it does solve the problem of these massive deployments that you want to break up into their uh individual components or individual features that you can then turn on and off but it's also a way to test those and see if they're working. So, you turn on a flag, but you only turn it on for maybe 10% I think as you alluded to earlier on, right, of your user base. And maybe that's a maybe that's a a progressive roll out or or maybe you turn it on just for your uh power users, for example, because they're going to give you more feedback and you're going to know faster whether that thing is working. So, there's all kinds of various segments you could turn on those flags for. And then, of course, you experiment on the other end. You set up an experiment and say, "Hey, is this impacting my checkout rate, right? Or or or my cart size or did I order an extra burger with my uh fast food order because I was testing this new way to present this data to you and to present an offer to you?" Right? There's all sorts of ways to use feature flags. And the biggest one we're hearing, of course, today is just dealing with the velocity of code being generated, right? There's so much code that's coming down the pike. And um you need a way to be able to deal with that. And the last thing you want to do is have, you know, 200 300% more features that you're releasing and still be in that old mode of releasing in one big chunk, you know, every other week or whatever. Like that's just it's just setting yourself up for some sort of a risk or some sort of sort of a vulnerability to creep in. >> Okay. So let's concrete that a bit because a feature release could be anything from we're going to change the entire language of the app or a foreign or programming or it could be something so small as like a different colored button. So can we get into some of the experimentation practically y'all did at Speedway or maybe your first experiment even how you figured out we're going to try that first. That's nerve-wracking. I remember the most successful experiment that I've done and that was we redid our product page. We had an old product page and like Aaron said, we were able to do a progressive roll out on that and this one's near and dear to my heart because I was leading it. I was I was running the project and I could see the results of the experiment positively influence business outcomes. And that was a fairly large experiment on the whole, but once it went out, we were able to do legitimately what you just said, Jennifer, we were able to do an experiment on the button color. Like there's a reason why Amazon's button color is orange. Orange and green have been measured to perform the most. Uh that's not an accident. They did a test on it and they did a test on the button color and it ended up successful for them. So you can be really nitpicky with with your experiments if you want to. >> So yeah, explain more about this. So you did a rebrand of your product page, I guess, and instead of So you still did like because often, >> especially developers get very precious about their websites. Um, let's call it what it is. And a lot of them built their own even though that's not their unique value proposition building website. uh respectfully of course say you got precious perhaps but a lot of times it's this really spoken to someone in content arduous process to get that up there and to make that change. So how did you did you make it in still like a switch from one to the other or did you progressively change? So it still was like a full rebrand launch but then when you launched you had all these toggles going. >> Yeah. So there was some leadup discovery work that we did before when we were on the old product page. We were able to go add feature flags to it and toggle certain features on and off to understand what the new pages needed like what's really important, what's really driving customer behavior. Then there was this big toggle where we went from the old to the new. We changed the tech stack underneath of it. But then once we deployed fully that new and measured it to have a positive outcome, there was the discovery work again where we were doing more smaller experimentations. We were working on stuff like recommendation sections. We were working on stuff like what placement of images is the most effective to to our customers. So there's kind of this small big small cadence that we use specifically for that project. But you can use any like you can put anything big or small behind a feature flag. Um they're very very flexible. >> Wow. Okay. >> And I would say one of the one of the lesser known I would I would just say it really does when you start looking at how to use feature flags, it really does open your mind a little bit. Um you know we're talking about basically you know testing features but you could also use it as part of a migration. I don't think that happened in this case, but you could certainly use feature flags as a way to move from on-prem to cloud or from one database to another database. uh one version of your application that's built on a particular uh language and structure, you know, um maybe you're using uh moving into containers and Kubernetes from more of a static inst there's just like all kinds of things once you start getting into this world of oh what can I control easily with this thing um that your mind starts kind of expanding the possibilities >> and honestly in the world of AI I and the cost of cloud etc. There's also people experimenting with moving workloads back they may be cloudnative workloads but back to onrem and figuring out which is more efficient more effective for a different workload. I'm very curious about this process. So did you have to hire people that were feature flag speakers or proficient? Did you expand your team because you had you were running these experiments? How did you perform training, onboarding into the feature management? >> We did not have to hire anybody. Our team picked it up. It was very intuitive. Yeah, coming from someone who does not have a technology background actually um I it's was very quick for me to pick up um figure out you know exactly I care more about the metrics of something exactly where to go how to look at those things um so even from like a non-technological aspect I would say it was just super quick to pick up >> and Jennifer just to say one other thing this kind of doubles down on something that you said too Aaron the feature flags themselves don't necessarily need to be experiments every time. There's such a thing as a safety feature flag. And I think that for developers, like having safe code is is incredibly important. So if a developer can see, oh, I can wrap this new feature that I'm building that I'm not going to have any test on. I have no intention on testing it, but it ensures the safety of the overall experience. and if something's broken on a deployment, I can just go turn it off immediately. I think that was a really good selling point for the engineering team, too. And it also helped the the adoption of it. I think maybe the more business background, the business-minded people really like the experiments and like the business outcome side of it. Like that that's makes intuitive sense to that crowd. But the safety and honestly reputation that comes with engineers is my code works and and that helps the adoption of feature flags from the from the technical side. >> That also closes that important feedback loop that's so that cognitive load that developers are holding and at a two week sprint you maybe didn't even remember what you did >> to then be able to fix it. But if you have an immediate feedback loop that closes that loop and it's also an adrenaline rush to be accepted or maybe not and have to fix and scramble it, but at least it comes back and you can fix it. >> Yeah. Certainly a lot better than rolling back a huge deployment with tens, 20s, however many features you deploy, right? Because that's the alternative. That's why it's kind of the safety blanket, if you will, on a feature-byfeature basis. Otherwise, you know, who knows? Is that roll back going to succeed? You know, all those features that you rolled out, whoever was exposed to those now doesn't have those anymore. Where if you're dealing with feature flags, it's you're dealing with it on a very, very tight window, very tight application where you can just say, "Oh, I'm just going to take that one thing that's failing and I'm going to turn it off. And by the way, I can give that functionality to my incident team." So they they're enabled to do it. I don't know if if at Speedway how you deal with that, but I mean it it does kind of democratize the ability to to handle when things go bad, so to speak. >> Yeah. And and one other thing that we can do just to double down on that point is you mentioned targeting before, Aaron. We can target ourselves first. We can do a production deployment and target, you know, our team, you know, and stakeholders directly and we can test it out on oursel in the production environment. I mean there's a joke about do you test in production? We do. We do test in production. We just don't test it on everybody. We we can specifically target ourselves and ensure that it's going to be good so no customers are ever going to have that poor experience if that were to happen. >> Do the business stakeholders or other stakeholders ever become a bottleneck in that situation? because then they're like, "Well, use it." And then they're like, "Well, you could change it a bit." >> Yeah, I'm sure that that does happen. You know, I don't think you're ever going to automate that away. Um, >> yeah, >> one thing it does allow us to do is if if that conversation does come up and a business stakeholder is like, "Hey, I'm not so sure about this." You can release to a segment of your customers. you can release to 5 10%. And that's when a test helps out a ton because if it's just my word against your word, that's that's one thing. But if we actually have data to back it up, that can kind of drive the conversation in a less political, more advantageous way. >> How does it work on the other side? Have you had, and of course nothing giving away the secret sauce, but have you had features that just didn't work? Like customers responded differently, stakeholders responded differently, and you just were surprised. How do you deal with that then? I think uh at that point, you know, it depends because for example, um we recently went to live with a fully landed cost shipping option for our international customers. Um and what we did is we started out about 5% just to make sure that you know we could see some orders get through full conception from uh order place to ship to getting through customs, make sure everything, you know, follows that path correctly. Um, and we actually found at about actually we went to 100%. So we got all the way to 100% found out that there was an issue once the packages were getting to customs. >> Um, they were being charged again by UPS. So what we were able to do is we turned that feature flag back off. Um, realized, okay, something's going on here. It gave us that time to be like, okay, we're stable. Obviously, we have it turned off so we know that um we're not offering fully landed cost anymore. Um so the expectation is correct on the customer's side of things, but then it gave us that moment to be like, okay, we're stable now. We can figure out what's going on, where the problem is. Is it on our end? Is it on UPS's end? Um so that was something that we were able to, you know, really quickly kind of dive into. Um and you know, this isn't the only one that we found out, oh no, we have to turn it off. you know, those things happen. Um, but it just really gives you that quick run back to be stable back to where you know things are okay and then give you that time to dive into what's going on. >> Yeah. And it's really easy to get pessimistic about experiments that didn't pan out because you're excited about them. >> But I think it's a win to learn. It's a good thing that you know not to do that. Um, learning in either direction, positive or negative, is a good thing. I would say maybe one of the harder points is if you do an experiment and you have inconclusive results. That's kind of a tough one. You have to choose a direction to go. Um but I'd rather run an experiment than not run one. >> Okay. Another use case brought from the audience is so thank you. Uh you can have feature flags for fallback for maintenance states to show a message. Also for areas of your code that you have a dependency on a third party such as figuring out state taxes to apply so you handle outages you can't control. I think that's great too because I think both examples that we just talked about are things are not as much as you want to experiment things are not always in your control. they may be in the world's largest shipping company's control that you are a very valued partner. So you have to figure that out and I think that's a really valid point. I'd love to get into the practicality now. Derek, do you want to tell us about how you actually go about what a feature flag looks like from the technical end? It sounds like Kelly on a more my style view. Yes. >> That it's business and that's important too. We know with platform engineering we need that single pane of glass so business stakeholders can see but I'm sure we have some technical listeners. So do you want to go into that how that works Derek? >> Yeah so how we implemented it was we have a a shared SDK library uh in our code in our service harness specifically. Um what that allows us to do is that shared code is more of a utility and we can call that from anywhere in our codebase. So that means you have this very thin call, this thin layer um that surrounds your feature or your third party. I was going to actually double down on that. We have a a feature flag that controls our warehouse right now when we have inclementware weather. If we have poor weather and we can't ship out, we can turn our warehouse off with a feature flag. >> Wow. >> Um, yeah. So, so you can have this be loadbearing in your overall operations. Uh, but yeah, it makes it very very easy. It's a thin call that you would do and then you just surround your code in an if statement. If this feature is on, do this. Else do the old thing. Okay. You had an image you wanted to share, right? Do you want to explain that? Bring that up. Like a good thing. I'm still amazed by that example, by the way. >> Yeah. Just to double down on that, like there's um permanent feature flags that we have. So, I kind of have this example that I shared with with Jennifer before this. Um this is an example of a really complicated permanent feature flag that we have. Um we have a very complicated recommendation solution that we have measured to be positive like in terms of business outcome. This has increased our revenue quite a bit. Uh what we've done is we've created just a JSON object that we've put inside of our feature flag configuration and it allows us to test very complicated uh examples against against uh just head-to-head. If you are blessed to have just copious amounts of traffic, you can actually run more than just AB. You can do an ABCD test if you want to and 25% of traffic goes to each and you can learn pretty rapidly. Um so this is an example here. It kind of does two things. We're able to do the experiment on head-to-head. Is it better to put mandatory Rex in front of consumable Rex or vice versa? That could be a hypothesis that we would test out. But say there's just a a simple text change that we want to make to our title. Uh we can just go in there and change the title without ever having to do a deployment. Uh it's instantly realized on the client. >> Wow. Fantastic. >> Yeah, I think that's an important um as people are learning how to do this or maybe trying to create their own DIY feature flag uh practice. Um, it it is kind of amazing because all this runs client side, how quickly you can change things. You change a feature flag and it's I I I can't say it's instantaneous from a mathematical perspective because that's not true. But when you're comparing it to to rolling out a new version of your app, let's say on your phone, hoping someone downloads it or whatever to get that upgrade. being able to just flip flip a toggle like literally have it appear within moments uh for your customer and then be able to see results come in almost immediately. It really is gamechanging. >> Well, thank you. I'm still obsessed with that idea of the weather shutting down the warehouse. That is the world we're entering >> for sure. >> Yeah. And it's super convenient. Like it used to be a panic, right? like we would, oh no, we have to go and do a deployment to turn off our warehouse and you want me to do that right now at 6:00 in the morning because it's snowing outside. Now it's just a feature flag. We did that that work once and we have the ability to toggle it whenever we want to. >> And you would call that like an evergreen feature flag. We had talked about feature flags that you reuse or keep running, but feature flag debt is like everything else. Everybody's got their zombie servers running. Everyone's got their zombie microservices running. Like how do you combat feature flag debt? How do you decide what is a permanent feature flag versus an experiment? When do you turn these off? >> Yeah, that's a great question. I would say our evergreen feature flags are ones that wrap third parties. Uh like the example given in chat, the ones that you know, our warehouse, that's an evergreen one. Our recommendations are going to be evergreen. Um, we've we've also got a evergreen feature flag that targets internal users. So, if you are a Speedway Motors employee, you can see certain uh data points on our site that other people won't. Mostly it's like search investigation. Um, so there's a couple dedicated ones that we have. The debt does pile up. Um, the best way to do like I would say the non- evergreen ones, the ones where you had an experiment and then you uh concluded the experiment, then you moved on with your life or you just had that safety feature flag where your code was behind a feature flag, you were able to deploy it, test it on yourself, not an experiment, and then just turn it on. Those feature flags do hang around. They do hang around. One thing that we have done to combat that is uh harness has an API and we have a task that will go and query harnesses API to see all of the feature flags that are outstanding and if it's not a dedicated evergreen feature flag if we haven't set that up as a as a tag for for permanent uh it will alert us. It'll say this thing's been on for over a month. Are you sure you want to keep it? And what is the risk of leaving feature flags on? >> Very low. Very low. I guess you could accidentally go back and turn it into the incorrect state and then you built code around it where now your feature flag being in the off state would not act nicely with the new code that you've written. Um >> that's just good code hygiene. Yeah. >> Yeah. the the feedback loop on that would be nice if we can go in and remove that from code more uh eagerly. Maybe that's an application for AI. >> Okay. >> Yeah. I I can't go into the details on that, Derek, on this one, but um >> uh stay all I can say is stay tuned for for just a little bit longer and uh we'll we'll we'll have a follow-up and see how that's going, I guess. >> But there are other mechanisms as well. I mean uh what Dererick is saying is is really popular actually because we do have a concept of pipelines and policies in place in the product. So you can set a step in your in your uh pipeline for example that will say uh go you know after this has hit 100% after this flag has been rolled out to 100% you have a step that's automated. it's part of your process that says go clean up this code, here's where it lives, and then make sure that gets deployed and updated as part of your next um deployment. So you absolutely can can at least automate it, make sure you have policies in place to enforce it so that people do go in and um clean that stuff up because you can imagine, you know, we're talking about individual use cases here, but those add up, especially for companies that have thousands and thousands of developers. Um that the amount of code that's that's kind of left hanging out there, it's important to go clean it up at the end of the day. >> And it's just going to be more code with AI. We're just going to have more of everything. It's going to amplify all the bad habits. So, let's get good hygiene and habits in place and avoid more debt. >> 100%. >> Love this audience question, so I pick it eagerly. How do you measure the impact of the feature that's toggled on? What metrics do you use? Do you pair with AB testing, etc.? Kelly, I think business probably super fun functionality, I guess, supersedes business, but business probably supersedes everything else in this, right? >> Yeah. I mean, there are metrics built into the harness app. So, um, you know, I think Erin touched on it just a little bit earlier, but um, as soon as you turn something on, it is immediately starting to read that data. um and it is comparing you know the certain percentage of what you have enabled or the certain segment of group of people that you have enabled to you know the off um control group right so um you know we try to run our experiments for at least two weeks just to have a good amount of data um but the metrics are definitely built right there in the harness app um and I think we can um you know really narrow in on the ones that we value the most um and look at them from that perspective effective. >> Yeah. And to follow that up, there's a lot of metrics that we have created specifically. >> There are very specific use cases like we have user added a year make model to their garage. That's only going to be applicable for an automotive retailer. But on top of that, you have your general e-commerce ones. Revenue, AOV, conversion rate. We're watching those like a hawk. Um the closer to the metal that you can get the better. So if you are like changing your search algorithm, you should test for search product engagement. You shouldn't look at the overall revenue increase. Although it is important to keep an eye on that. It's just a couple steps removed from the feature that you are changing. Uh I would encourage I guess to have both. There's no reason to not have both as key metrics. Uh but definitely look as close as you can to the feature that you are changing. >> Absolutely. >> And AOV for those like myself not in e-commerce stands for average order value says the Google AI results. So I hope that's >> that is correct. I'm sorry. >> No, it's fine. I was like should I know this? Should I I don't work in e-commerce. It's okay. I don't know this to say online. Uh, >> threeletter acronyms. >> So many threeletter acronyms. >> Business and tech together. It's worse than the UN. We have another great audience question I want to get to before we wind down. How do you manage testing when you don't really know which other flags will be enabled when you release? Oh, and I guess this comes when we talk about the scale too of a bank, which is the largest tech companies for a lot of people. How I'm understanding that question is the test that you're writing like the code feature has multiple uh feature flags enabling it or maybe there's like a nesting there's some sort of dependency between the two. Um on that I would say communication is your friend. Make sure that you're talking to the person that is also in the area that you're working on. Um I would say the very best way to do it just from a statistical standpoint would be to have every condition covered. So make a truth table off off on on off on on um and figure out which one wins in those. Now it could be the case that some of those are not viable where there the dependency is such that it would cause an error state if one's on and the other's off. Um, if you can anticipate that, make sure it doesn't happen in code. If if that error case is met, uh, just do the default. Just do what was happening before and then alert on it so that the team can can go in and and fix that issue. Okay, that's definitely a case for avoiding your future flag debt going up too much and you know keep good hygiene as well so that doesn't happen because I honestly hadn't thought of that but it's dolls nesting dolls all the way down sometimes or turtles all the way down. So >> yeah. >> Yeah. Go. >> Oh, I was just going to say like we have a smaller team so it hasn't happened all that much. we're kind of in a whole bunch of different areas or one team's over here and another team's over over there. So, it doesn't happen all that much in practice as far as I know. Um, but for a larger team, I could definitely see that happening. >> Aaron, have you heard anything at some of the enterprises? Um I mean it is something you have to watch out for. But I think the if you look for um uh best practices for example or maturity of companies that are doing this well they try not to let that happen. Right? You want to be testing something very very specific and then you want to be doing quick experiments on whether that moves the needle or not and then moving on from there uh when you can. So you definitely don't want to be combining or even in your code having multiple triggers if you can help it because it just makes it more difficult obviously I guess to to figure out what what is actually going on. So the more specific you can be um and deliberate you can be and then the more experiments you can run um is going to lead to better results down the road. >> Yes. And and I will say just to add on uh Derek, correct me if I'm wrong at any point here, we also have a connection uh with harness in one of our Slack channels. Um so if anyone is changing a future flag or an experiment um it will automatically send a message to that Slack channel with the environment what was changed um etc. So all of our developers have access to that. Now that's on you know the engineers to be reading that but it is just another way to like hey this is changing. Um, so just that that there's more communication out there as well. >> Yeah. And and of course, like I said before, if you're if you're using policies and pipelines in the product, you can also set that as a check right before it goes live. So you can c you can catch things and you can do hooks into Slack and all kinds of other applications as well. So yeah, >> and when it comes down to it, we're talking about technology, but people and processes are what tend to be the most important part of any technical change. So as we wind down and since stating goals make them more likely to come true, where do you hope that your organization goes next on your future flag journey, how you're going to leverage them next? That's that's a good question. >> Is AI going to be involved or no? >> Most likely, yes. Um, we are very heavily experimenting with AI at this point. So, you know, we're always looking for ways to evolve. I would say we're very much proactive in many different aspects of looking into, you know, how can we always be better? Um, AI is definitely on that. Uh, Derek, do you have any specifics there maybe? >> Yeah. Uh, one annoying experience that I've had with AI is like the configuration on the requests to an AI agent. Uh, you can change from chat GPT to the newest version that Anthropic has out and they've been updating versions fairly often. One convenient thing that I've done is I've made a feature flag, a permanent one that just has what model I want to use in the prompt text so that I don't have to do a deployment every time I change my integration with AI. I can just go and change the model in the configuration. >> Well, within the feature flag, you then update that notification. So it's on your list like hey when this update happens every other week probably right now at the speed we're going at you set it it's like on your list of things to do when you're of your own hygiene your own best practices with AI. >> Uh if there's not a problem I won't go and change it. But when I'm finding that one certain model is not doing what I want it to do and there's another one out there, maybe it maybe I'm changing from like a a thinking model that has the ability to scrape websites to just a nano or something for cost reasons. I can I can do that from a feature flag via that configuration. Very similar to the recommendation example I showed earlier. Um it makes it a lot easier so that I don't have to go and deploy all my services to up update my codes integration with a model. >> Yeah. And this goes down a pretty big rabbit hole of token use and all kinds of other things, right? Where you could very easily switch processes from one model to another, from one prompt to another um using feature flags um in a way that you know could could actually impact the bottom line. It could make your results better or worse. I mean, it has all sorts of implications to how you're using AI in your in your environment right now. That's a great example. >> So important. I wish you luck in your fle feature flag progressive delivery experimentation. Somebody had to make it illiterative to make me more challenged as a host. Uh, and I'm so grateful to the audience. First of all, you've brought great questions. There is a free harness demo that you can get in the resources so you can play around with feature flags because experimentation starts with experimentation and we know developers especially want to get their hands on it. But I love that also the business stakeholders can too. And thank you so much to Derek, Kelly and Aaron and have a great day all. Thank you. >> Yeah, thank you very much. It's been great. >> Thank you. Hey, hey, hey., Case Study: Feature Flags IRL, YouTube, 01kqzea95vxnx6jkpn222zt80p.