From Concept to Cultural Revolution  ## Metadata - Author: [[featureflagsbook.com]] - Full Title: From Concept to Cultural Revolution - Category: #articles - Document Note: # Title: Feature Flags: From Concept to Cultural Revolution - - - tags: #FeatureFlags #SoftwareDevelopment #ProductDevelopment Author: Ben Nadel Source: [Incident Commander](www.incident-commander.com) Reference: Not Applicable date created: 2023-12-27T09:30 - - - ## Table of Contents: - Caveat Emptor - Of Outages And Incidents - The Status Quo - Feature Flags, An Introduction - Key Terms and Concepts - Going Deep on Feature Flag Targeting - The User Experience (UX) of Feature Flag Targeting - Types of Feature Flags - Life-Cycle of a Feature Flag - Use Cases - Bridging the Sophistication Gap - Life Without Automated Testing - Coding Defensively - Server-Side vs. Client-Side - KISS: Keep It Super Simple - Ownership Boundaries - Logs, Metrics, and Feature Flags - The Hidden Cost of Feature Flags - Not Everything Can Be Feature Flagged - Build vs. Buy - Track Actions, Not Feature Flag State - A Cultural Revolution - People Like Us Do Things Like This - The Democratization of Product Design - Co-creating the MVP (Minimum Viable Product) - An Opinionated Guide To Pull-Requests (PRs) - Dynamic Code, Dynamic Teams - Removing the Cost of Context Switching - Measuring Team Productivity - The Goal Gradient - I Eat, I Sleep, I Feature Flag ## Index Creation: - Feature Flags - Outages - Incidents - Cultural Revolution - Product Development - Waterfall Methodology ## 📝 Summary: "Feature Flags: From Concept to Cultural Revolution" by Ben Nadel explores the concept of feature flags and their profound impact on software development. The author emphasizes that feature flags are not just a technical tool but can drive a cultural shift within product development teams. Feature flags enable controlled flow within software, allowing teams to respond to issues proactively and improve product quality. The book encourages readers to be critical and open-minded, adapting the concepts to their unique situations. ## 📖 Main Content: ### 🗒️ Notes - Feature flags are a versatile tool for controlling flow within software. - They have a cultural impact, transforming the way teams work and deliver value to customers. - Crashing production was considered a rite of passage in the early days but led to fear and overthinking in engineering. - Incident Commander training and certification helped manage outages but lacked consistency in communication. - Feature flags brought stability and improved productivity while reducing incidents. - The Waterfall Methodology can lead to delays and misunderstandings in product development. - Feature flags enable dynamic functionality and more responsive development. ## 📖 References: - Not Applicable ## 🗣️ Paraphrase: 1. Original: "Feature flags are a deceptively simple concept. It can be hard to understand the extent of the impact they have on your team because the implications aren't just technical." Paraphrased: "Feature flags may seem simple, but their impact on teams goes beyond the technical aspects, making them challenging to grasp fully." 2. Original: "Quality increases when you go faster. Productivity increases when you work smaller. Innovation and discovery happen at the edge, in that unpredictable, heady space where Product and Customer meet." Paraphrased: "Speed enhances quality, working on a smaller scale boosts productivity, and innovation thrives at the unpredictable intersection of Product and Customer." ## 🧪 Hypotheses: 1. Hypothesis: Implementing feature flags can lead to improved product quality and team productivity. 2. Hypothesis: The Waterfall Methodology may result in delays and misunderstandings in software development projects. ### 📊 Methodology: 1. Research the adoption of feature flags in different software development teams and analyze their impact on product quality and team performance. 2. Conduct surveys and interviews with software development teams using the Waterfall Methodology to identify common challenges and delays in the development process. ### 💡 Implications: 1. Implementing feature flags as a cultural change can enhance product development processes and customer satisfaction. 2. Reconsidering traditional development methodologies like Waterfall may lead to more agile and responsive software development. ### 🔮 Future Research: 1. Investigate the long-term effects of feature flags on team culture and productivity. 2. Explore alternative methodologies that promote both quality and speed in software development. ### ❓ Implications: 1. Implementing feature flags can lead to a cultural shift in product development, improving collaboration and responsiveness. 2. Relying solely on the Waterfall Methodology may hinder innovation and lead to misunderstandings in project requirements. ### ✅ Logical validity: The article presents a logical progression of ideas, starting with the concept of feature flags and their impact on teams. The author provides personal experiences and insights to support their claims. However, it's important to note that the article is based on the author's perspective and may not apply universally to all software development scenarios. ## Conclusion: "Feature Flags: From Concept to Cultural Revolution" by Ben Nadel highlights the transformative power of feature flags in software development. It emphasizes their role in both technical and cultural aspects of product development, encouraging readers to adapt and explore these concepts in their own contexts. Feature flags offer a path to improved quality, productivity, and innovation in software development, making them a valuable tool for modern development teams. - Document Tags: [[developer experience]] [[featuredflagsbook]] [[product development]] - Summary: Feature flags are a powerful tool that can bring about cultural change in product development. They enable the decoupling of feature release from deployment, allowing for safer and more incremental rollouts. Feature flags require a robust runtime functionality and can be used for targeting specific user experiences. However, they also add complexity to codebases and require careful management to prevent unnecessary complexity. Not all changes can or should be put behind feature flags, and it's important to consider the trade-offs and hidden costs associated with their implementation. Building a custom feature flags system may not be the most efficient use of time, as feature flags are deceptively simple but can have a profound impact on product development workflows. - URL: https://featureflagsbook.com/  ### Feature Flags: From Concept to Cultural Revolution by [Ben Nadel](https://www.bennadel.com/) #### [#](https://featureflagsbook.com/#TableofContents)Table of Contents 1. [Caveat Emptor](https://featureflagsbook.com/#CaveatEmptor) 2. [Of Outages And Incidents](https://featureflagsbook.com/#OfOutagesAndIncidents) 3. [The Status Quo](https://featureflagsbook.com/#TheStatusQuo) 4. [Feature Flags, An Introduction](https://featureflagsbook.com/#FeatureFlagsAnIntroduction) 5. [Key Terms and Concepts](https://featureflagsbook.com/#KeyTermsandConcepts) 6. [Going Deep on Feature Flag Targeting](https://featureflagsbook.com/#GoingDeeponFeatureFlagTargeting) 7. [The User Experience (UX) of Feature Flag Targeting](https://featureflagsbook.com/#TheUserExperienceUXofFeatureFlagTargeting) 8. [Types of Feature Flags](https://featureflagsbook.com/#TypesofFeatureFlags) 9. [Life-Cycle of a Feature Flag](https://featureflagsbook.com/#LifeCycleofaFeatureFlag) 10. [Use Cases](https://featureflagsbook.com/#UseCases) 11. [Bridging the Sophistication Gap](https://featureflagsbook.com/#BridgingtheSophisticationGap) 12. [Life Without Automated Testing](https://featureflagsbook.com/#LifeWithoutAutomatedTesting) 13. [Coding Defensively](https://featureflagsbook.com/#CodingDefensively) 14. [Server-Side vs. Client-Side](https://featureflagsbook.com/#ServerSidevsClientSide) 15. [KISS: Keep It Super Simple](https://featureflagsbook.com/#KISSKeepItSuperSimple) 16. [Ownership Boundaries](https://featureflagsbook.com/#OwnershipBoundaries) 17. [Logs, Metrics, and Feature Flags](https://featureflagsbook.com/#LogsMetricsandFeatureFlags) 18. [The Hidden Cost of Feature Flags](https://featureflagsbook.com/#TheHiddenCostofFeatureFlags) 19. [Not Everything Can Be Feature Flagged](https://featureflagsbook.com/#NotEverythingCanBeFeatureFlagged) 20. [Build vs. Buy](https://featureflagsbook.com/#BuildvsBuy) 21. [Track Actions, Not Feature Flag State](https://featureflagsbook.com/#TrackActionsNotFeatureFlagState) 22. [A Cultural Revolution](https://featureflagsbook.com/#ACulturalRevolution) 23. [People Like Us Do Things Like This](https://featureflagsbook.com/#PeopleLikeUsDoThingsLikeThis) 24. [The Democratization of Product Design](https://featureflagsbook.com/#TheDemocratizationofProductDesign) 25. [Co-creating the MVP (Minimum Viable Product)](https://featureflagsbook.com/#CocreatingtheMVPMinimumViableProduct) 26. [An Opinionated Guide To Pull-Requests (PRs)](https://featureflagsbook.com/#AnOpinionatedGuideToPullRequestsPRs) 27. [Dynamic Code, Dynamic Teams](https://featureflagsbook.com/#DynamicCodeDynamicTeams) 28. [Removing the Cost of Context Switching](https://featureflagsbook.com/#RemovingtheCostofContextSwitching) 29. [Measuring Team Productivity](https://featureflagsbook.com/#MeasuringTeamProductivity) 30. [The Goal Gradient](https://featureflagsbook.com/#TheGoalGradient) 31. [I Eat, I Sleep, I Feature Flag](https://featureflagsbook.com/#IEatISleepIFeatureFlag) #### [#](https://featureflagsbook.com/#CaveatEmptor)Caveat Emptor I have opinions. Often, these opinions are strong; and, in most cases, these opinions are strongly held. But, they are just *my* opinions. In this book, I will speak with an air of authority because I believe deeply in what I am saying based on what I have seen: a team transformed. But, what I am saying is based on my own experience, context, and organizational constraints. What I say may not always apply perfectly to you and your situation. You are a discerning, creative person. You are here because you are enrolled in the work of building better products; of building more effective teams; and, of delivering more value to your customers. Do not let that creativity take a back seat as you read this book. Be critical, but open; question my assertions, but do not dismiss them out of hand. Feature flags are a deceptively simple concept. It can be hard to understand the extent of the impact they have on your team because the implications aren't *just* technical. If all you learn from this book is how to use feature flags as a means to control flow within your software, this book will be worth reading. However, the *true value* of what I'm sharing here is in the *holistic cultural change* that feature flags can bring to every part of your product development life-cycle. This book does not represent an all-or-nothing approach to product development. But, I do believe that the more you take from this book, the more you will get. #### [#](https://featureflagsbook.com/#OfOutagesAndIncidents)Of Outages And Incidents I used to tell my people: *"You're not really a member of the team until you crash production"*. In the early days of the company, crashing production—or, at least, significantly degrading it—was nothing short of inevitable. And so, instead of wielding each outage as a weapon, I treated it like a rite of passage: an attempt to create a safe space in which my people would learn about and become accustomed to our product. Engineers *need* to ship code. This isn't a company mandate, it's a matter of self-actualization. Pushing code to production benefits us—and our mental wellbeing—just as much as it benefits our customers. But, when deployments become fraught, engineers become fearful. They begin to overthink their code and under-deliver on their commitments. This wasn't good for the product. And, it certainly wasn't good for the team. Not to mention the fact that it created an unhealthy tension between our Executive Leadership Team (ELT) and—well—everyone else. The more people we added to our engineering staff, the worse this problem became. Poorly architected code with no discernible boundaries soon lead to tightly coupled code that no one wanted to touch let alone maintain. Code changed in one part of the application often had a surprising impact over in another part of the application. The whole situation felt chaotic and unstoppable. The best we could do at the time was prepare people for the trauma. We implemented an "IC Certification" program. The "IC"—or, Incident Commander—was responsible for getting the right people in the (Slack) room; and then, liaising between the triage team and the rest of the organization. To become IC certified, you had to be trained by an existing IC. You had to run through a mock incident and demonstrate that you understood: * How to identify the relevant teams. * How to page the relevant teams, waking them up if necessary. * How to start the "war room" Zoom call. * How to effectively communicate the status of the outage (and estimated time to remediation). This IC training and certification program was mandatory for all engineers. The issues were very real and very urgent; and, we needed *everyone* to be ready. The certified ICs were good at communicating status, but each IC had their own style—their own way of writing and presenting updates to the team. This lead to a lot of fumbling and inconsistency, which ultimately distracted us from the goal at hand. To address this, I built a small utility that brought order to the output: Incident Commander ([www.incident-commander.com](https://www.incident-commander.com/)). This online tool provided ICs with a way to translate status updates into pre-formatted Slack message which the IC could then copy-and-paste into the `#incident` channel. As a team, we became quite adept at responding to each incident. And, in those early days, this coalescing around the chaos formed a camaraderie that bound us together. Even years later, I still look back on those Zoom calls with a fondness for how close I felt to those that were there fighting alongside me. But, the kindness and compassion we offered each other meant nothing to our customers. The incidental joy we felt from a shared struggle was no comfort to those that were paying us to provide a stable product. Our CTO (Chief Technical Officer) understood this. He never measure downtime in *minutes* - he measured it in *lost opportunities*. He painted the picture of customers, victimized by our negligence: > "People don't care about SLOs (Service Level Objective) and SLAs (Service Level Agreements). 30-minutes of downtime isn't much on balance. But, when it takes place during your pitch meeting and prevents you from securing a life-changing round of Venture Capital, 30-minutes means the difference between a path forward and a dead-end." > > He put a Root Cause Analysis (RCA) process in place, and *personally reviewed* every write-up. Remediating an incident was only a minor victory, preventing the *next incident* was the real goal. Each RCA included a technical discussion about what happened, how we became aware of the problem, how we identified the root cause, and the steps we intended to take in order to prevent it from occurring again. The RCA process—and the Quality Assurance Items (QAI) that they generated—did create a more stable *platform*. But, a platform is merely the foundational world that lives below the surface of the product. Most of the work that we were doing took place above the platform, in the ever-evolving user-facing feature-set. A stable platform is a necessity. But, as the platform stabilized, the RCA process began to see a diminishing return on investment (ROI). Even as the platform improved, the outages continued to happen. In a last ditch effort to effect better outcomes, a Change Control Board (CCB) was put in place. A CCB is a group of subject matter experts (SME) that *must review and approve* all changes being made to the product. A Change Control Board is the antithesis of worker autonomy, it is the antithesis of productivity. A Change Control Board says, *"we don't pay you to think."* A Change Control Board says, *"we don't trust you to use your best judgement."* If workers yearn to find fulfillment in self-direction, enhanced responsibility, and a sense of purpose, the Change Control Board seeks to strip responsibility and treat workers as nothing more than mindless resources. And yet, with the choke-hold of the CCB in place, the incidents continued. After working on and maintaining the same product for over a decade, I have the luxury of hindsight and experience. I can see what we did right and what we did wrong. The time and effort we put into improving the underlying platform was time well spent. Without a solid foundation on which to build, nothing much else matters. Our mistake was in trying to create a predictable outcome for the product. We slowed down the design process in hopes that every single pixel was in the correct location. We slowed down the deployment pipeline in hopes that every single edge-case and feature had been fully-implemented and tweaked to perfection. We thought that we could increase quality and productivity by *slowing everything down*. But, the opposite is true. Quality increases when you *go faster*. Productivity increases when you *work smaller*. Innovation and discovery happen *at the edge*, in that unpredictable, heady space where Product and Customer meet. Eventually, we learned these lessons. Outages and incidents went from a daily occurrence to a weekly occurrence to a rarity. At the same time, productivity went up, team morale was boosted, and our paying customers finally started to see the value we promised them. And, none of it would have been possible without feature flags. Feature flags *changed everything*. #### [#](https://featureflagsbook.com/#TheStatusQuo)The Status Quo There's no "one way" for organizations to build and deploy a product. Even a single engineer will use different techniques in different contexts. When I'm at work, for example, I use a Slack-based chatbot to trigger new deployments; which, in turn, communicates with Kubernetes; which, in turn, executes an incremental rollout of new Docker containers. But, in my personal life—on side projects—I *still* use FTP (File Transfer Protocol) in order to *manually sync* files between my local development environment and my VPS (Virtual Private Server). No given approach to web development is inherently "right" or "wrong". Some approaches do have advantages. But, everything is a matter of nuance; and, every approach is based on some set of constraints and trade-offs. At work, I get to use a *relatively sophisticated* deployment pipeline because I stand on the shoulders of giants who came before me. But, when I'm on my own, I don't have the ability to create that level of automation and orchestration. Though many different approaches exist, most build and deployment strategies do have one thing in common: when code is deployed to a production server, users start to consume it. Add a new item to your navigation and—*immediately*—users start to click it. Change a database query and—*immediately*—users start to execute it. Refactor a workflow and—*immediately*—users start to explore it. We're decades beyond the days of shipping floppy disks and CD-Roms; but, most of us still inhabit a state in which *deploying code* and *releasing code* are *the same thing*. Having users come to us—and our servers—gives us the ability to respond to issues more proactively; but, fundamentally, we're still delivering a "static product" to our customers. To operate within this limitation, teams will oftentimes commit temporary logic to their control flow in order to negotiate application access. For example, a team may only allow certain parts of an application to be accessed if: * The user is connecting though the VPN (Virtual Private Network). * The request is coming from a set of allow-listed IP addresses. * The authenticated user has an internal company email address. * The incoming request contains a secret HTTP `Cookie` value. * The incoming URL contains a special query-string parameter. * The HTTP `User-Agent` contains a magic string. I've used many of these techniques myself. And, they all work; but, they are all subpar. Yes, they *do allow* internal users to inspect a feature prior to its release; but, they offer little else in terms of dynamic functionality. Plus, exposing the gated code to a wider audience requires changing the code and deploying it to production. Which, conversely, means that *re-gating* the code—such as in the case of a major bug or incident—requires the code to be updated or reverted and re-deployed. Going beyond their hard-coded nature, these techniques often treat deployment as an afterthought. Meaning, they are typically implemented *only after* a feature has been completed and is now ready for review. This implies that the feature has been articulated and developed in relative isolation. This is referred to as the "Waterfall Methodology". Waterfall Methodology comprises a set of product development stages that get executed in linear sequence: 1. Analysis and requirements gathering. 2. Graphic design and prototyping. 3. Implementation. 4. Testing and quality assurance (QA). 5. Deployment to production. 6. Maintenance. Each one of these stages is intended to be completed in turn, with the outputs from one stage acting as the inputs to the next stage. At first blush, the Waterfall Methodology is attractive because it looks to create structure and predictability. But, this is *mostly* an illusion. In the *best case* scenario, the timeline for such a project is *grossly underestimated*. In the *worst case* scenario, the engineering team builds the *wrong product*. > **Anecdote**: Years ago, I worked at a company that applied the waterfall methodology. On one particular project for a data-management tool, the team carried out the requirements gathering, did some graphic design, and then entered the implementation phase as per usual. But, building the product took a lot longer than anticipated (which is standard even in the "best case" scenario). And, naturally, the client was very upset about the slipping release date. > > Finally, after many delays and many heated phone calls and much triaging, the team performed their "big reveal" to the client. And, after walking through the product, the client remarked, *"This isn't at all what I asked for."* > > It turns out, there was a large understanding gap in the requirements gathering phase. This understanding gap was then baked into the design process which was subsequently baked into the engineering process. > > The client was furious about the loss of time and money (with nothing to show for it). The engineers were furious because they felt that the client hadn't been forthcoming during the analysis phase (classic victim blaming). And, of course, product management was furious because the project failed and reflected poorly on our firm. > > This isn't a black swan event. Many of us in product development have similar stories. As an industry, we *mostly* agree that the Waterfall Methodology is doomed; and, that "Agile Methodology" is the preferred approach. The Agile Methodology takes the Waterfall Methodology, shrinks it down in scope, and repeats it over and over again until the product is completed. The cornerstone of Agile is a strong emphasis on **"People over processes and tools"** using a **continual feedback loop**: * Build small. * Show your work. * Refactor based on feedback. * Repeat. If our team had been using an Agile Methodology (in the earlier anecdote), success *would* have been achieved. The client *would* have seen early-on that the product was moving in the wrong direction and they would have told us. In turn, the design and engineering teams *would* have changed course and adapted to the emergent requirements. And, ultimately, all stakeholders *would* have been happy with the outcome. The Agile Methodology is fantastic! At least, it is within a "greenfield" project—one in which no prior art exists. But, as soon as you release a product to the customers, all *subsequent changes* are being made within a "brownfield" project. This is where things get tricky. Or, expensive. Oftentimes both. Agile Methodology has advantages over Waterfall Methodology; but, eventually, both approaches run into the same problem: **deploying code to production is *still* a dangerous proposition**. And so, the fear creeps in; and soon, even "agile" teams with the best intentions start to fall back into old waterfall tendencies. In hopes of leaving nothing to chance, the design process becomes endless. Lack of trust in the engineering team leads to a longer, more tedious QA period. Paranoia about outages means no more deploying on Fridays (or, perhaps, with even *much less* frequency). Test coverage percentage becomes a target. Expensive staging environments are created and immediately fall out-of-sync. A product manager creates a deployment checklist and arbitrarily makes "load testing" a blocking step. The whole system—the whole process—starts to creak and moan under the pressure (of time, of cost, of expectation) until, at some point, someone makes the joke: > Work would be great if it weren't for all the customers. > > In a business where customer empathy builds the foundation of all great products, wanting to work *without customers* becomes the sign of something truly toxic: cultural death. This isn't leadership's fault. Or the fault of the engineers or of the managers or of the designers. This doesn't happen because the wrong technology stack was chosen or the wrong management methodology was applied. It has nothing to do with your people working in-office or being remote. This happens because a *small seed of fear* takes root. And then grows and grows and grows until it subsumes the entire organization. Fear erodes trust. And, without trust we don't feel safe. And, if we don't feel safe, the only motivation we have left is that of *self-preservation*. This sounds dire (and it is); but, it isn't without hope. All we have to do is address the underlying fear and everything else will eventually fall into place. This transformation may not come quick and it may not come easy; but, it can be done; and, it starts with feature flags. #### [#](https://featureflagsbook.com/#FeatureFlagsAnIntroduction)Feature Flags, An Introduction I've been working in the web development industry since 1999; and, before 2015, I'd never heard the term, "feature flag" (or "feature toggle", or "feature switch"). When my Director of Product—[Christopher Andersson](https://www.linkedin.com/in/christopherandersson/)—pulled me aside and suggested that feature flags might help us with our company's "downtime problem", I didn't know what he was talking about. Cut to me in 2023—after 8-years of trial, error, and experimentation—and I *can't imagine* building another product or platform *without* feature flags. They have become a critical part of my success. I hold feature flags in the same light as I do **Logs** and **Metrics**: the essential services on which *all product performance and stability* are built. But, it wasn't love at first sight. In fact, when Christopher Andersson described feature flags to me in 2015, I didn't see the value. After all, a feature flag is "just" an `if` statement: ``` if ( featureIsEnabled() ) { // Execute NEW logic. } else { // Execute OLD logic. } ``` I already had control flow that looked like this in my applications (see **[The Status Quo](https://featureflagsbook.com/#TheStatusQuo)**); as such, I didn't understand why adding *yet another* dependency to our tech-stack would make any difference in our code (let alone have a positive impact on our downtime). What I failed to see was the fundamental difference *underlying* the two techniques. In my approach, changing the behavior of the `if` statement meant updating the code and re-deploying it to production. But, in the case of feature flags, changing the behavior of the `if` statement meant *flipping a switch*. That's it. No code updates. No deployments. No latency. No waiting. This is the **magic of feature flags**: having the ability to **dynamically change** the **behavior of your application** at **runtime**. This is what sets feature flags apart from environment variables, build flags, and any other type of deploy-time or dev-time setting. To stress this point: if you can't dynamically change the behavior of your application *without touching* the code *or* the servers, you're *not* using "feature flags". The dynamic runtime nature isn't a nice-to-have, it's *how* feature flags bring both the *psychological safety* and the *democratizing of power and creativity* to your organization. This dynamic nature means that in one moment, our feature flag settings can look like this: User interface (UI) control for a product feature flag currently in the "Off" state. Which means that our application's control flow operates like this: ``` if ( featureIsEnabled() /* false */ ) { // ... dormant code ... } else { // This code is executing! } ``` The `featureIsEnabled()` function is currently returning `false`, directing all incoming traffic through the `else` block. Then, if we *flip the switch* on in the next moment, our feature flag settings look like this: User interface (UI) control for a product feature flag currently in the "On" state. And, our application's control flow operates like this: ``` if ( featureIsEnabled() /* true */ ) { // This code is executing! } else { // ... dormant code ... } ``` Instantly—or thereabouts—the `featureIsEnabled()` function starts returning `true`; and, the incoming traffic is diverted away from the `else` block and into the `if` block, changing the behavior of our application in real-time. But, turning a feature flag *on* is only half the story. It's equally important that—at any moment—a feature flag can be turned *off*. Which means that, should we need to (**in case of emergency**), we can instantly disable the feature flag: User interface (UI) control for a product feature flag currently in the "Off" state. Which will *immediately revert* the application's control flow back to its previous state: ``` if ( featureIsEnabled() /* false */ ) { // ... dormant code ... } else { // This code is executing (again)! } ``` Even with the illustration above, this is still a rather abstract concept. To convey the power of feature flags more concretely, let's dip-down into the real-world use-case that opened my eyes to the possibilities: refactoring a SQL database query. The efficiency of a SQL query changes over the lifetime of a product. As the number of rows increases and the access patterns evolve, some SQL queries start to slow down. This is why [database index design is just as much art as it is science](https://www.bennadel.com/blog/3467-the-not-so-dark-art-of-designing-database-indexes-reflections-from-an-average-software-engineer.htm). Traditionally, a refactoring of this type might involve running an `EXPLAIN` query locally, looking at the query plan bottlenecks, and then rewriting or breaking the SQL apart in order to better leverage existing table indices. The query code, once updated, is then deployed to the production server. And, what the you *hope to see* is a latency graph that looks like this: Line graph showing SQL query latency over time. The latency has a marked improvement after the code is deployed. In this case, the SQL refactoring was effective in bringing the query latency times back down. But, this is the *best case scenario*. In the *worst case scenario*, deploying the refactored query leads to a latency graph that looks more like this: Line graph showing SQL query latency over time. The latency increases greatly after the code is deployed. In this case, something went terribly wrong. For any number of reasons, the SQL query that performed well in your local development environment does not perform well in production. The query latency rockets upward, consuming most of the database's available CPU. This, in turn, slows down *all queries* executing against the database. Which, in turn, leads to a spike in concurrent queries. Which, in turn, crashes the database. If you see this scenario unfolding in your metrics, you might try to roll-back the deployment; or perhaps, try to revert the code and redeploy it. But, in either case, it's a race against time. Pulling down images, spinning up new nodes, warming up containers, starting applications, running builds, executing unit tests: it all takes time—time that you *don't have*. Now, imagine that, instead of completely refactoring your code and deploying it, you design an *alternate* SQL query and gate it behind a feature flag. Code in your data-access layer could look like this: ``` public array function generateReport( userID ) { if ( featureIsEnabled() ) { return( getData_withOptimization( userID ) ); } return( getData( userID ) ); } ``` In this approach, both the existing SQL query and the *alternate* SQL query get "*deployed*" to production. However, the *alternate* SQL query won't be "*released*" to the users until the feature flag is enabled. And, at that point, the `if` statement will short-circuit the control flow and all new requests will use the optimized SQL query. With this feature flag in place, the *worst case scenario* now looks rather tame: Line graph showing SQL query latency over time. The latency increases greatly after the feature is released; but, returns to normal after the feature is disabled. The same unexpected SQL performance problem exists in this scenario. However, the outcome is very different. First, notice that the "deployment" itself had no effect on the latency of the query. That's because the alternate SQL query was deployed in a *dormant* state behind the feature flag. Then, the feature flag was enabled, causing traffic to route through the alternate SQL query. At this point, that latency starts to go up; but, instead of the database crashing, the feature flag is *turned off*, immediately re-gating the code and diverting traffic back to the original SQL query. **You just avoided an outage**. The dynamic runtime capability of your feature flag gave you the power to react without delay, before the database—and your application—became overwhelmed and unresponsive. Are you beginning to see the possibilities? Knowing that you can disable a feature flag in case of emergency is empowering. This alone creates a huge amount of psychological safety. But, it's only the beginning. Even better is to completely avoid an emergency in the first place. And, to do that, we have to dive deeper into the robust runtime functionality of feature flags. In the previous thought experiment, our feature flag was either *entirely on* or *entirely off*. This is a vast improvement over the status quo; but, this isn't really how feature flags get applied. Instead, a feature flag is normally "rolled-out" *incrementally* in order to minimize risk. But, before we can think incrementally, we have to understand both "targeting" and "variants". Targeting is the act of identifying which users will receive a given a variant. And, a variant is the value returned by evaluating a feature flag in the context of a given request. To help elucidate these concepts, let's take our very first `if` statement and factor-out the `featureIsEnabled()` call. This will help separate the feature flag evaluation from the subsequent control flow: ``` var booleanVariant = featureIsEnabled(); if ( booleanVariant == true ) { // Execute NEW logic. } else { // Execute OLD logic. } ``` In this example, our feature flag represents a Boolean value, which implicitly has two possible variants: `true` and `false`. Targeting for this feature flag then means figuring out which requests receive the `true` variant and which requests receive the `false` variant. Flow diagram showing the relationship between requests, feature flag targeting, and variant selection. Boolean feature flags are, by far, the most common. However, a feature flag can represent *any* kind of data type: Strings, Numbers, Dates, JSON (JavaScript Object Notation), etc. These non-Boolean data types may compose any number of variants and unlock all manner of compelling functionality. But, for the moment, let's stick to our Booleans. Targeting—the act of funneling requests into a specific variant—requires us to provide identifying information as part of the feature flag evaluation. The "right" identifying information is going to be context-specific; but, I find that "User ID" and "User Email" are a great place to start: ``` var booleanVariant = featureIsEnabled( userID = request.user.id, userEmail = request.user.email ); if ( booleanVariant == true ) { // Execute NEW logic. } else { // Execute OLD logic. } ``` Once we incorporate this identifying information into our feature flag evaluation, we can begin to differentiate one request from another. And this is where things start to get exciting. Instead of our feature flag being *entirely on* for *all users*, perhaps we only want it to be on for an allow-listed set of User IDs. One implementation of such a `featureIsEnabled()` function might look like this: ``` public boolean function featureIsEnabled( numeric userID = 0, string userEmail = "" ) { switch ( userID ) { case 1: case 2: case 3: case 4: return( true ); break; default: return( false ); break; } } ``` Or, perhaps we only want the feature flag to be on for users with an internal company email address: ``` public boolean function featureIsEnabled( numeric userID = 0, string userEmail = "" ) { if ( userEmail contains "@bennadel.com" ) { return( true ); } return( false ); } ``` Or, perhaps we only want the feature flag to be enabled for a *small percentage* of users: ``` public boolean function featureIsEnabled( numeric userID = 0, string userEmail = "" ) { var userPercentile = ( userID % 100 ); if ( userPercentile <= 5 ) { return( true ); } return( false ); } ``` In this case, we're using the Modulo operator to *consistently translate* the User ID into a numeric value. This numeric value gives us a way to *consistently map* users onto a percentile: each additional "remainder" represents an additional 1% of users. Here, we're enabling our feature flag for a *consistently-segmented* 5% of users. We can even combine several of these targeting concepts at once in order to exert even more granular control. Imagine that we want to target internal company users only; *and*, of those targeted users, only enable the feature for 25% of them: ``` public boolean function featureIsEnabled( numeric userID = 0, string userEmail = "" ) { if ( userEmail contains "@bennadel.com" ) { var userPercentile = ( userID % 100 ); if ( userPercentile <= 25 ) { return( true ); } } return( false ); } ``` User targeting, combined with a %-based rollout, is an incredibly powerful part of the feature flag workflow. Now, instead of enabling a (potentially) risky feature for *all users* at one time, imagine a much more **graduated rollout** using feature flags: 1. Deploy dormant code to production servers. 2. Enable feature flag for ***your* user ID**. 3. Test feature *in production*. 4. Discover a bug. 5. Fix bug and redeploy code (still only active for your user). 6. Examine error logs. 7. Enable feature flag for ***internal* company users**. 8. Examine error logs and metrics. 9. Discover bug(s). 10. Fix bug(s) and redeploy code (still only active for internal company users). 11. Enable feature flag for **10%** of *all* users. 12. Examine error logs and metrics. 13. Enable feature flag for **25%** of *all* users. 14. Examine error logs and metrics. 15. Enable feature flag for **50%** of *all* users. 16. Examine error logs and metrics. 17. Enable feature flag for **75%** of *all* users. 18. Examine error logs and metrics. 19. Enable feature flag for ***all*** users. 20. Celebrate! Few deployments will need this much rigor. But, when the risk level is high, the control is there; and, much of the risk associated with your deployment will be mitigated. **Are you beginning to see the possibilities?** So far, for the sake of simplicity, I've been hard-coding the "dynamic" logic within our `featureIsEnabled()` function. But, in order to facilitate the graduated deployment outlined above, this encapsulated logic must also be dynamic. This is, perhaps, the most elusive part of the feature flags mental model. The feature flag evaluation process is powered by a "rules engine". You provide inputs, identifying the request context (ex, "User ID" and "User Email"). And, the feature flag service then applies its rules to your inputs and returns a variant. There is nothing random about this process—it is "pure", deterministic, and repeatable. The *same rules* applied to the *same inputs* will always result in the *same variant output*. Therefore, when we talk about the "dynamic runtime nature" of feature flags, it is in fact the **rules**, within the **rules engine**, that are actually dynamic. Consider an earlier version of our `featureIsEnabled()` function that ran against the `userID`: ``` public boolean function featureIsEnabled( numeric userID = 0, string userEmail = "" ) { switch ( userID ) { case 1: case 2: case 3: case 4: return( true ); break; default: return( false ); break; } } ``` Instead of a `switch` statement, let's refactor this function to use a data structure that reads a bit more like a "rule configuration": ``` public boolean function featureIsEnabled( numeric userID = 0, string userEmail = "" ) { var rule = { input: "userID", operator: "IsOneOf", values: [ 1, 2, 3, 4 ], variant: true }; if ( ( rule.operator == "IsOneOf" ) && rule.values.contains( arguments[ rule.input ] ) ) { return( rule.variant ); } return( false ); } ``` The outcome here is exactly the same, but the mechanics have changed. We're still taking the `userID` and we're still looking for it within a set of defined values; but, the static values and the resultant variant have been *pull out* of the evaluation logic. At this point, we can move the *rule definition* out of the `featureIsEnabled()` function and into its own function, `getRuleDefinition()`: ``` public boolean function featureIsEnabled( numeric userID = 0, string userEmail = "" ) { var rule = getRuleDefinition(); if ( ( rule.operator == "IsOneOf" ) && rule.values.contains( arguments[ rule.input ] ) ) { return( rule.variant ); } return( false ); } public struct function getRuleDefinition() { return({ input: "userID", operator: "IsOneOf", values: [ 1, 2, 3, 4 ], variant: true }); } ``` Here, we've completely decoupled the *consumption* of our feature flag rule from the *definition* of our feature flag rule. Which means, if we needed to change the outcome of the `featureIsEnabled()` call, we wouldn't change the logic within the `featureIsEnabled()` code at all. Instead, we would update the `getRuleDefinition()`. But, everything is *still hard-coded*. In order to make our feature flag system dynamic, we need to replace the hard-coded data-structure with *something* like: * A database query. * A Redis `GET` command. * A reference to a shared in-memory cache (being updated in the background). Which creates an application architecture like this:  Distributed application architecture showing that the feature flag administration makes real-time changes to the running application rules cache. The implementation details will depend on your chosen solution. But, each approach reduces down to the same set of concepts: a feature flag administration system that can **update the active rules** being used within the **feature flag rules engine** that is **currently operating** in a given environment. This is what makes the dynamic runtime behavior possible. It may seem that integrating feature flags into your application logic includes a lot of low-level complexity. But, don't be put-off by this—you don't *actually* have to know how the *rules engine works* in order to extract the value. I only step down into the weeds here because having *even a cursory understanding* of the low-level mechanics can make it much easier to understand how feature flags fit into your product development ecosystem. The reality is, any feature flags implementation that you choose will abstract-away most of the complexity that we've discussed. All of the variants and the user-targeting and %-based rollout configuration will be moved out of your application into the feature flags administration, leaving you with *relatively simple* code that looks like this: ``` var useNewWorkflow = featureFlags.getBooleanVariant( feature = "new-workflow-optimization", context = { userID: request.user.id, userEmail: request.user.email } ); if ( useNewWorkflow ) { // Execute NEW logic. } else { // Execute OLD logic. } ``` This alone will have a meaningful impact on your product stability and uptime. But, it's only the beginning—the knock-on effects of a feature-flag-based development workflow will bleed into your entire organization. It will transform the way you think about product development; it will transform the way you interact with customers; and, it will transform the very nature of your company culture. #### [#](https://featureflagsbook.com/#KeyTermsandConcepts)Key Terms and Concepts Feature flags enable a new way of thinking about product development. This introduces some new concepts; and, adds more nuance to some existing ideas. As such, it's important to define—and perhaps redefined—some key terms that we use in this book: ##### Deploying Deploying is the act of pushing code up to a production server. Deployed code is *not necessarily* "live" code. Meaning, deployed code isn't *necessarily* being executed by your users - it may be sitting there, on the server, in a dormant state. "Deployed" refers only to the *location* of the code, not to its participation within the application control flow. A helpful analogy might be that of "commented out" code. If you deploy code that is commented out, that code is living "in production"; but, no user is actually executing it. Similarly, if you deploy code behind a feature flag and no users are being targeted, then no user is actually executing that deployed code. ##### Releasing Releasing is the act of exposing deployed code and functionality to your users. Before the advent of feature flags, "Deploying" code and "Releasing" code were generally the *same thing*. With feature flags, however, these two actions can now be decoupled and controlled independently. ##### Feature Flag A feature flag is a *named gating mechanism* for some portion of your code. A feature flag typically composes an identifier (ex, `new-checkout-workflow`), a type (ex, Boolean), a set of variants (ex, `true` and `false`), a series of targeting rules, and a rollout strategy. Some of these details may vary depending on your feature flag implementation. ##### Variant A variant is one of the distinct values returned when evaluating a feature flag in the context of a given request. All feature flags have *at least* two variants—with *only* a *single* variant, you can't create a *dynamic runtime behavior*. Each variant value is an instance of the Type represented by the feature flag. For example, a Boolean-based feature flag can only return one of two finite values: `true` or `false`. On the other hand, a Number-based feature flag can return *any number* of variants between `-Infinity` and `Infinity` (depending on how numbers are represented in your programming language). That said, at any given moment, the variants aggregated within a feature flag are *finite*, *typically* few, and *predictable*. Entropy has no place in a feature flag workflow. ##### Targeting Targeting is the mechanism that determines which feature flag variant is served to a given user. Targeting rules include both assertions about the requesting user and a rollout strategy. Targeting rules may include *positive assertions*, such as "the user role is Admin"; and, it may include *negative assertions*, such as "the user is *not* on a Free plan". Compound rules can be created by `AND`ing and `OR`ing multiple assertions together. The conditions within the targeting rules can be changed over time; however, at any given moment, the evaluation of the decision tree is repeatable and deterministic. Meaning, the same user will always receive the same variant when applying the same inputs to the same rules. ##### Rollout Rollout is an overloaded term in the context of feature flags. When we are discussing a feature flag's configuration, the rollout is the strategy that determines which variant is served to a set of targeted users. This is often expressed in terms of percentage. For example, with a Boolean-based feature flag, the rollout strategy may assign the `true` variant to 10% of targeted users and the `false` variant to 90% of targeted users. When not discussing a feature flag's configuration, the term rollout is generally meant to describe the *timeline* over which a feature will be enabled within the product. There are two types of rollouts: Immediate and Gradual. With an immediate rollout, the deployed code is released to all users at the same time. With a gradual rollout, the deployed code is released to an *increasing* number of users *over time*. So, for example, you may start by rolling a feature out to a small group of Beta-testers. Then, once the feature has preliminary success, you roll it out to 5% of the general audience; and then 20%; and 50%; and so on, until the *deployed* code has been *released* to *all* users. ##### Roll-Back Just as with "rollout", roll-back is another overloaded term in the context of feature flags. When we are discussing a feature flag's configuration, rolling back means reverting a recent configuration change. For example, if a targeted set of users is configured to receive the `true` variant of a Boolean-based feature flag, rolling back the feature flag would mean updating the configuration to serve the `false` variant to the same set of users. When not discussing a feature flag's configuration, the term rolling back is generally meant to mean removing code from a production server. Before the advent of feature flags, if newly-deployed code caused a production incident, the code was then "rolled back", meaning that the new code was removed and the previous version of the application code was put back into production. ##### User In this book, I often refer to "users" as the receiving end of feature flags. But, this is only a helpful metaphor as we often think about our products in terms of customer access. In reality, a feature flag system doesn't know anything about "users" - it only knows about "inputs". Most of the time, those inputs will be based on the requesting user. But, they don't have to be. We'll get into this more within our use-cases chapter, but feature flag inputs can be based on *any meaningful identifier*. For example, we can use a "server name" to affect platform-level features. Or, we can use a *static value* (such as `app`) to apply the feature flag state to all requests uniformly. ##### Progressive Delivery This is the combination of two concepts: *deploying* a feature incrementally and *releasing* a feature incrementally. This is, eventually, the natural state for teams that *lean into* a feature-flag-based workflow. This becomes "the way" you develop products. ##### Environment An environment is the application context in which a feature flag configuration is defined. A given feature flag can be consumed across multiple environments. But, each environment is configured independently such that a feature flag enabled in a "local development" environment has no bearing on *the same feature flag* in a "production" environment. ##### Feature Flag Administration This is the application that your Developers, Product Managers, Designers, Data Scientists, etc. will use to create, configure, update, release, and roll-back feature flags. This application is generally separated out from your "product application"; but, it doesn't *have* to be. If you are *buying* a feature flag SaaS (Software as a Service) offering, your vendor will be building, hosting, and maintaining this administration module for you. #### [#](https://featureflagsbook.com/#GoingDeeponFeatureFlagTargeting)Going Deep on Feature Flag Targeting > **Note**: If the targeting concepts that we discussed earlier make sense to you, feel free to skip this chapter. We're about to dip down into a more philosophical and technical view in an effort to better illustrate the mechanics of targeting. This is often the hardest part of feature flags to understand; so, I think it warrants a deeper exploration. > > As web application developers, we generally communicate with the database anytime we need to gather information about the current request. These external calls are fast; but, they do carry an overhead. And, the more calls that we make to the database while processing a request, the more *latency* we add to the response-time. In accordance with this, feature flag implementations work using an **in-memory "rules engine"** which allows feature flag state to be "queried" without having to communicate with an external storage provider. This keeps the processing time *blazing fast*! So fast, in fact, that you *should* be able to query for feature flag state as many times as you need to without ever having to worry about latency. > **Aside**: Obviously, *all processing* adds *some* degree of latency; but, the in-memory feature flag processing—when compared to a network call—should be considered negligible. > > That said, shifting from a database-access mindset to a rules-engine mindset can present a stumbling block for the uninitiated. At first, it may be unclear as to why you need anything more than a "User ID" (for example) in order to do all the necessary *user targeting*. After all, in a traditional server-side context, said "User ID" opens the door to any number of database records that can then be fed into any number of control flow conditions. But, when you *can't* go back to the database to get more information, all targeting must be done using the information at hand. Which means, in order to target based on a given data-point, said data-point must be *provided to* the feature flag state negotiation process. I find that a "Pure Function" provides a helpful analogy. A pure function, will always result in the *same output* when invoked using the *same inputs*. This is because the result of the pure function is based solely on the inputs (and whatever internal logic exists). No external calls are made within a pure function. No side-effects are generated within a pure function. Unless there is a bug, a pure function will never error. It has no network calls that can fail. It has no file I/O calls that might lack the proper permissions. It has nothing that incurs a variable latency. Consider this function: ``` function calc( a, b ) { return( a * b + 1 ); } ``` If you invoke the `calc()` function with arguments `(2,4)` you will always get the result, `9`. Even if you invoke this function 1,000 times in a row, you will get `9` on every single execution. This is because the result is based entirely on the inputs and the inputs aren't changing. If you wanted the result of a `(2,4)` invocation to change, you'd have to change the underlying logic embedded within the function itself. For example, you might change the `+1` to a `+2`. This would change the *absolute* result of the invocation; but, the *relative* result of all invocations *going forward* would be the same. Feature flag targeting works the same way. If you want to target based on an *email address* (for example), you'd have to invoke the targeting mechanism using the *email address* as an input. And, the same email input will always result in the same targeting output; *unless*—and until—you change the logic embedded within the targeting system; and, that's where the "rules engine" come into play. The rules engine represents that *embedded logic*. Only, unlike a *pure function*, a feature flag system allows us to dynamically change the embedded logic *at runtime* (without the deployment of code). ##### A *Simplistic* Feature Flag Targeting Implementation Sometimes, a picture is worth a 1,000 words. And, sometimes, an engineer won't truly understand how something works until they pull back the covers and look at the code. As such, I want us to step-through the building of a **completely lacking**, **overly simplistic**, **toy version** of a feature flags system. What we look at here *will not* represent production-grade quality by any means; but, I *hope* that it contains detail enough to shed light on this confusing topic. As in the previous chapters, we're going to define our feature flags using a dictionary of simple data structures. Each structure contains the following information: * The variants that can be served by the feature flag targeting. * The default rollout strategy. * An optional, single rule that can target a subset of requests and apply a different rollout strategy. A Boolean feature flag, with no optional rule, looks like this: ``` { variants: [ false, true ], distribution: [ 100, 0 ] } ``` The fist array (`variants`) defines the collection of valid variants that can be returned from the feature flag evaluation process. The second array (`distribution`) defines the *weighted distribution* of the corresponding variants. In this case, the `100` means that 100% of users will receive the `false` variant (and the `0` means that 0% of users will receive the `true` variant). If we wanted to start rolling this feature out slowly, we could change the `distribution` property: ``` { variants: [ false, true ], distribution: [ 90, 10 ] } ``` Here, we're saying that 90% of the users will receive the first variant (`false`) and 10% of the users will receive the second variant (`true`). The number of items in the `variants` array must match the number of items in the `distribution` array (as the latter corresponds to the weighted distribution of the former). And, in the case of a Boolean feature flag, only 2 items make sense. But, if we were to serve up non-Boolean data, these arrays can be longer. For example, if we wanted to implement dynamic logging in a production application, we could create a String-based feature flag that serves the "minimum log level" to be aggregated: ``` { variants: [ "error", "warn", "info", "debug", "trace" ], distribution: [ 100, 0, 0, 0, 0 ] } ``` Here, the `variants` represent the possible log levels emitted by the application. And, in this case, 100% of all requests will only aggregate `error` level entries and higher. If we were in the middle of an incident and wanted to turn on lower-level logging, we might move the `100` from the 1st index (`error`) to the 4th index (`debug`). Then, all requests coming into the production application would start aggregating `error`, `warn`, `info`, and `debug` level log entries: ``` { variants: [ "error", "warn", "info", "debug", "trace" ], distribution: [ 0, 0, 0, 100, 0 ] } ``` Of course, turning `debug` level logging on for an *entire* application at one time could both overwhelm the system and lead to a non-trivial cost increase. Instead, we can try to enable low-level logging for only *5% of users* and hope that the *slightly increased* logging provides enough insight: ``` { variants: [ "error", "warn", "info", "debug", "trace" ], distribution: [ 95, 0, 0, 5, 0 ] } ``` Or, maybe we only want to turn `debug` level logging on for *specific users*, such as our internal Product Engineers. That's where our "rule" property comes into play. The "rule" property allows us to target a subset of users and apply a different distribution of the same `variants` just to them. A rule contains an operator, two operands, and the weighted distribution that will be used *if* the request passes the operator assertion. So, if we want to turn low-level logging on for our own developers, our rule could use the `IsOneOf` operator and test the incoming "User ID" against a set of *known developer IDs*: ``` { variants: [ "error", "warn", "info", "debug", "trace" ], distribution: [ 100, 0, 0, 0, 0 ], rule: { operator: "IsOneOf", input: "UserID", values: [ 1, 16, 34, 2009 ], distribution: [ 0, 0, 0, 100, 0 ] } } ``` With this configuration, the feature flag serves the `error` log level to 100% of users *by default*. However, if the requesting `UserID` "is one of" the given set of values, `[1, 16, 34, 2009]`, then 100% of *that subset* of users will receive the `debug` log level. We can make this targeting *even more flexible* by allowing for an *array of rules* in which each rule targets a different cohort of users. Instead of turning `debug` logging on for *just* our engineers, perhaps we *also* want to turn it on for the customer reporting the bug: ``` { variants: [ "error", "warn", "info", "debug", "trace" ], distribution: [ 100, 0, 0, 0, 0 ], rules: [ { operator: "IsOneOf", input: "UserID", values: [ 1, 16, 34, 2009 ], distribution: [ 0, 0, 0, 100, 0 ] }, { operator: "IsOneOf", input: "CompanySubdomain", values: [ "acme-inc" ], distribution: [ 0, 0, 0, 100, 0 ] } ] } ``` Using these simple constructs alone—variants, distributions, rules, and operators—we have everything we need to create a powerful targeting system. Now that we have a sense of how the underlying configuration works, we need to write the code that maps application requests onto feature flag variants. And, even though we are keeping this implementation as simple as possible, we still benefit from the coding best practices like decomposition, coding to interfaces, and encapsulation. To that end, I want to model the concepts of the "operator" and the "distribution" as separate components. This will help keep all the code in our top-level "Features" component at the same level of abstraction. Let's start with the implementation of our operators. Each *type* of operator (ex, "IsOneOf", "StartsWith", "EndsWith", etc) will be modeled as a separate component; but, each of these components will adhere to the same interface which exposes a single method: `test( contextValue, values ) : Boolean` The `contextValue` parameter is the value associated with the incoming request. In our previous configuration, this would be the "User ID" of the authenticated user. The `values` parameter is the array of *configured values* against which the `contextValue` is being tested. In our previous configuration, this would be the collection of known Production Engineer IDs. In order to implement our `IsOneOf` operator, all we have to do is loop over the `values` collection and perform an equality check against the `contextValue`: ``` component hint="IsOneOf operator." { public boolean function test( required any contextValue, required array values ) { for ( var value in values ) { if ( value == contextValue ) { return( true ); } } return( false ); } } ``` The interface for each operator is the same; but, of course, the implementation details will vary. As another example, let's look at the `EndsWith` operator which checks the `contextValue` against a collection of *suffix* `values`: ``` component hint="EndsWith operator." { public boolean function test( required any contextValue, required array values ) { for ( var value in values ) { if ( right( contextValue, len( value ) ) == value ) { return( true ); } } return( false ); } } ``` Using this simple interface, we can create any number of operators: Equality testing, prefix testing, suffix testing, substring testing, date comparisons, Regular Expression (RegEx) pattern matching, range testing, etc. Each `test()` implementation is a "pure function" that operates solely on its own invocation parameters. The distribution concept is a bit trickier to implement. In our configuration object, we're defining the variant distribution as a *weighted distribution* based on a percent allocation. When we have this configuration: ``` { variants: [ false, true ], distribution: [ 90, 10 ] } ``` ... it means that the `false` variant should be assigned 90% of requests and the `true` variant should be assigned to 10% of requests. A simplistic way to implement this logic is to create an array with 100 items, wherein 90 of the indices contain the first variant (`false`) and 10 of the indices contain the second variant (`true`). You can conceptualize this array using the spread operator and some pseudo-code: ``` [ ...arrayRepeat( false, 90 ), ...arrayRepeat( true, 10 ) ] ``` If we then *randomly* choose from this 100-item array, 90% of the random choices *should* result in `false` and 10% of the random choices *should* result in `true`. Of course, nothing about a feature flags system is "random". In fact, randomness would completely negate the value-add of gating code behind a feature flag. Instead, what we need is a way to *consistently map* the incoming request onto one of the 100 indices. And for this, we will need to use a context property that *consistently and repeatedly represents* the same "user". In our implementation, we're going to *assume* that this context property is **numeric** so that we can use the **Modulo operator** in order to easily bucket an *opened-ended* number of values into a 100-index distribution. Our `WeightedDistribution` component will then take this property, construct the 100-index allocation, and return the variant chosen by mapping the property onto the allocation index: ``` component hint="WeightedRollout distribution." { public any function getVariant( required numeric contextKey, required array variants, required array distribution ) { var allocatedVariants = []; distribution.each( ( percent, variantIndex ) => { for ( i = 1 ; i <= percent ; i++ ) { allocatedVariants.append( variants[ variantIndex ] ); } } ); var keyIndex = ( ( ( contextKey - 1 ) % 100 ) + 1 ); return( allocatedVariants[ keyIndex ] ); } } ``` In this code, our `allocatedVariants` is the 100-item array that contains the weighted distribution of the given variants. We then take the `contextKey`—our numeric "User ID"—consistently map it onto a `1..100` range (inclusive) and return the corresponding variant. This way, the **same user** is always assigned the **same variant** when given the **same distribution**. At this point, we have our operator implementations and our weighted distribution implementation. Now, let's compose them together into a coherent feature flags system. And, to configure this feature flags system, we'll use a dictionary of individual feature configuration objects. This is the data structure that we will "administer" in order to create **dynamic runtime behaviors** within our application. Each feature within this data structure must have a *unique name* so that our application code has something to reference when incorporating feature flags into the request control flow. Example: ``` { "new-checkout-process": { variants: [ false, true ], distribution: [ 50, 50 ] }, "internal-admin-tools": { variants: [ false, true ], distribution: [ 100, 0 ], rule: { operator: "IsOneOf", input: "userType", values: [ "Admin" ], distribution: [ 0, 100 ] } } } ``` Remember that our feature flag evaluation process is acting like a "pure function". Which means that when we check the state of a feature flag in the context of a given request, we have to *pass in* the "request context" data such that our "pure function" has all the inputs it needs in order to process the evaluation. At a minimum, checking the feature flag state requires the name of the feature in question (ex, `new-checkout-process`) and the unique ID representing the user. This unique ID is what we use to perform the Modulo calculation within our `WeightedRollout` component. In our request context object, we're going to refer to this unique ID as `key`: ``` features.getVariant( "new-checkout-process", { key: request.user.id } ); ``` Since our feature flags implementation uses a weighted distribution of variants, *all* calls to the `.getVariant()` method *must* include the `key` property. And, if we check a feature flag that includes a rule that operates on an additional property, we *must* include that additional property in the context object. So, if we wanted to check the `internal-admin-tools` feature, which has a rule that operates on `userType`, we'd have to pass `userType` in as *an additional* part of the context object in the `.getVariant()` invocation: ``` features.getVariant( "internal-admin-tools", { key: request.user.id, userType: request.user.type } ); ``` With our feature flags configuration object and our request context invocation object, the `.getVariant()` method implementation ends up doing little more than plucking-properties and orchestrating function calls. Here's our `Features` component—the `config` object being passed into the `init()` constructor is our feature flags configuration data structure: ``` component hint="Features" { public void function init( required struct config ) { variables.config = arguments.config; variables.operators = { IsOneOf: new operators.IsOneOf(), NotIsOneOf: new operators.NotIsOneOf(), StartsWith: new operators.StartsWith(), EndsWith: new operators.EndsWith() }; variables.weightedRollout = new rollouts.WeightedRollout(); } // --- // PUBLIC METHODS. // --- public any function getVariant( required string featureName, required struct context ) { var feature = config[ featureName ]; var distribution = feature.distribution; // The rules property is OPTIONAL. If it doesn't exist, // fallback to an array so we can assume its existence. var rules = ( feature.rules ?: [] ); // For simplicity, we want to allow for an OPTIONAL // single rule property as well. In that case, we'll // override the rules collection. if ( feature.keyExists( "rule" ) ) { rules = [ feature.rule ]; } for ( var rule in rules ) { var operator = operators[ rule.operator ]; var contextValue = context[ rule.input ]; var values = rule.values; // If the operator test "passes", override the // distribution for the current feature flag evaluation. if ( operator.test( contextValue, values ) ) { distribution = rule.distribution; // The first rule that matches, wins. No need // to check any other rules. break; } } var contextKey = context.key; var variants = feature.variants; var variant = weightedRollout.getVariant( contextKey, variants, distribution ); return( variant ); } } ``` As you can see, when we initialize our `Features` component, we cache our configuration—the dictionary of named feature flags—we instantiate and cache our operators, and we instantiate and cache our weighted distribution rollout strategy. Our `.getVariant()` method then weaves these parts together when the application needs to check the state of a *given feature* in the context of a *given request*. If we wanted to change the feature flags configuration, we *could* re-instantiate the `Features` component with a new data structure. But, most applications *cache components* as part of the bootstrapping processing. As such, in order to enable that sweet, sweet **dynamic runtime behavior**, we need to add a way to hot-swap the configuration within an existing `Features` instance. And, to do that, we can add a `.setConfig()` method: ``` component hint="Features" { // ... truncated code ... public void function setConfig( required struct config ) { variables.config = arguments.config; } } ``` Now, we can change the feature flags configuration *at runtime* which can change the outcome of a feature flag evaluation: ``` ALWAYS_FIRST_VARIANT = [ 100, 0 ]; ALWAYS_SECOND_VARIANT = [ 0, 100 ]; features = new lib.Features({ "new-checkout-process": { variants: [ false, true ], distribution: ALWAYS_FIRST_VARIANT } }); // Outputs FALSE since all users get the FALSE // variant under the current configuration. writeOutput( features.getVariant( "new-checkout-process", { key: 12345 } ) ); // Change the configuration at RUNTIME. features.setConfig({ "new-checkout-process": { variants: [ false, true ], distribution: ALWAYS_SECOND_VARIANT } }); // Outputs TRUE since all users get the TRUE // variant under the new configuration. writeOutput( features.getVariant( "new-checkout-process", { key: 12345 } ) ); ``` This code gives us the output: `false true`. The first call to the `.getVariant()` method uses the *initial* configuration, resulting in `false`. Then, the second call to `.getVariant()` uses the *runtime-updated* configuration, resulting in `true`. And just like that, we have a **functioning feature flags implementation**! It has *no* data validation and *no* type safety; and, it's missing the majority of functionality that you'd find in a production-grade implementation; but, I'm hoping that by diving into the mechanics of variants, user targeting, and percent-based distribution, you'll start to form a much stronger mental model for how feature flags can be woven into the fabric of your application. #### [#](https://featureflagsbook.com/#TheUserExperienceUXofFeatureFlagTargeting)The User Experience (UX) of Feature Flag Targeting There are two groups of people affected by feature flag targeting: the people who use your product; and, the people who manage your feature flag system. Most of this book is about the former—I'd like to take a moment and talk about the latter. When you first experiment with feature flags, you're almost certainly going to start by using your database "primary keys" as the value to target. After all, these database keys are the inputs we use in our application request processing; so, it's only natural to extend that mindset into the realm of feature flag targeting. Which means, if you wanted to target an "organization" (ie, a "semantic container" for a cohort of users), you might use the organization's database ID: ``` { variants: [ false, true ], distribution: [ 100, 0 ], rule: { operator: "IsOneOf", input: "companyID", values: [ 1728 ], distribution: [ 0, 100 ] } } ``` This will absolutely work. **However**, it turns out that numbers have very little inherent meaning within your application. So, when a feature flag administrator goes to configure a feature flag and sees something to the effect of: `values: [ 1728, 9034, 10348 ]` ... it's not immediately obvious as to *who* is being targeted within your application. Or, if the targeted companies are even still relevant. Since the feature flag administration system is *likely to be* a SaaS offering, "drift" will occur. That is, your feature flags system will eventually reference data that no longer exists within your production application. If, on the other hand, a feature flag administrator went to configure a feature flag and saw **subdomains** instead of **IDs**: `values: [ "acme-inc", "ben-inc", "testing" ]` ... it would be obvious which users are in the intended target audience. And, it would be easier to determine, at a glance, whether or not those targeted cohorts are still relevant. Whenever possible, I suggest that you use a **human-friendly String** as your means of targeting. For users, this might be their email address. And, for organizations, this might be their subdomain. Of course, it's all a balancing act. While a database ID is unlikely to ever change, email addresses and subdomains *do* change (occasionally). And, when they do change, the feature flag targeting rules will have to be updated to reflect this change. That said, I would choose an intelligible value over a static value. #### [#](https://featureflagsbook.com/#TypesofFeatureFlags)Types of Feature Flags Feature flags vary in both the type of variants that they serve and in the way they are expected to operate within your application. Short-lived, Boolean-based features flags are almost certainly going to be the first type that you use. Their on/off state is easy to reason about; and, it closely aligns with the `if` statements that we commonly use in our application control flow. But, feature flags can be used—and abused—in many different ways. ##### Data Types Any type of data that can be *serialized for storage* can be used in a feature flag. We think about feature flags as being a runtime concern—and they are primarily; but, they also have to be persisted across application restarts and rolling deployments. As such, we can only use data that serializes and deserializes without ceremony. This rules-out complex runtime objects such as instantiated Classes and Function references. But, it also adds challenges to *relatively* simple data-types such a Date/Time values. We use Date/Time values so often in programming that we are lulled into viewing them as native data-types. But, they are, in fact, instances of Classes that represent date and time using some underlying value translation (such as a Unix Timestamp). > **Aside**: "Sets" and "Maps" are common Object-like data-types that are hard to represent in a serialized format. And, therefore, don't work well as feature flag variants. > > If we wanted to use Date/Time values as feature flag variants, we'd have to configure them in a serialized format and then handle the deserialization in our application code. For example, we could serialize a Date/Time value using an **ISO 8601 formatted String**: `"2023-10-01T00:00:00.000Z"` Or, we could serialize it as a **Number** that represents the milliseconds since January 1, 1970, UTC (the UNIX epoch): `1696118400000` Neither of these values represents a Date/Time value implicitly—they only become a Date/Time value when we use them with intent. And, a feature flags system will *not know* your intent. Which means, we have to perform the manual translation on our end (in the application code): ``` // Parse ISO string into JavaScript Date. Date.parse( "2023-10-01T00:00:00.000Z" ) // Instantiate new JavaScript Date from milliseconds. new Date( 1696118400000 ) ``` It's best to think of variant data as anything that can be represented as **JSON (JavaScript Object Notation)**. In fact, we can build an entire feature flags implementation using *just* the JSON data type. But, since JSON is a *ceremonious* format, most feature flag implementations allow for other data types—such as Strings, Numbers, and Booleans—to be used. These, however, are likely little more than *syntactic sugar* on top of an underlying JSON storage value. Meaning, when you go to store this **String** value as a variant *without quotes*: `info` ... the feature flags system is likely serializing it by wrapping the value in quotes (for you) and then serializing it at as **JSON**: `serializeJson( "info" )` These developer ergonomic efficiencies are part of the value-add of a production-grade feature flags SaaS offering. With that said, Boolean-type feature flags will work in the vast majority of your use-cases. Strings, Numbers, and JSON data-types are absolutely a value-add; but, they really shine brightest in more specialized, more nuances use-cases. ##### Conceptual Types Feature flags fall into two *conceptual* types. Mechanically, these two types are exactly the same—they differ only in *how long* they are *intended* to remain embedded within your application logic. I call these two types: * **Product**: relatively short-lived feature flags. * **Operations**: long-lived (possibly forever) feature flags. > **Note**: These are the terms that I use personally - they are not industry standards. > > A **Product feature flag** is intended to be used during the development of a new *product* feature. And, when said feature is enabled for all customers (and has had time to "soak" in production), this feature flag *should be removed* from the application. Product feature flags should have a finite timeline. **Operations feature flags**, on the other hand, are tied to the ongoing operation of the application itself and have no inherent timeline. The "log level" feature flag that we discussed earlier is the perfect example of an *operational* feature flag—it will continue to add value for as long as the application is generating log entries. As such, there is no intention to remove this feature flag from the code. Given that these two *conceptual* types are *mechanically* identical, the value-add in considering this classification comes down to *developer ergonomics*. Feature flags add complexity to a codebase. As such, it is paramount that we do everything that we can in order to keep the code easy to understand. And, to this end, I strongly encourage you to create divergent naming patterns for product feature flags and operations feature flags. This way, when a developer is looking at the code, it becomes *immediately clear* as to the *intended* role of a given feature flag. I've taken to prefixing my product feature flags with `product-` and my operations feature flags with `OPERATIONS-`. Furthermore, my product feature flags are also infixed with a ticket number, such that I can easily tie a feature back to an epic or user story. A product feature flag that represents a new checkout process might be named: `product-GROWTH-1234-checkout-process` ... where `GROWTH-1234` is the corresponding ticket number. And, an operations feature flag that represents log level aggregation settings might be named: `OPERATIONS-minimum-log-level` This simple naming convention, on top of a conceptual dichotomy, can go a long way in terms of reducing the cognitive load imposed on the people working in the product. It can also make it clear which feature flags are *intended* to be there; and, which may be forgotten relics, begging for removal and simplification. > **Note**: For the sake of brevity, I *do not* following this naming convention within the examples laid-out in the book. > > Since this classification represents an arbitrary difference, not a technical one, I find it helpful to consider the *consumer* of the feature flag. If the *intended consumer* of the feature flag is *the user*—and, enabling the feature flag will alter the user experience in some way—it's a "product" feature flag. If, however, the *intended consumer* is the *application*, it's an "operations" feature flag (even if that sometimes means a user experience will be affected as a byproduct of feature flag state). Sometimes, a "product" feature flag will never be released to 100% of users; and, therefore, will never be removed from the application. In such a scenario, I still consider the feature flag to be of the "product" type; especially if future removal of the targeted users could one day merit removal of the feature flag. #### [#](https://featureflagsbook.com/#LifeCycleofaFeatureFlag)Life-Cycle of a Feature Flag If you're used to building an *entire* feature locally and then deploying it *all at once* to production, it can be hard to understand where to *start* with feature flags. In fact, your current development practices may be so deeply ingrained that the value-add of feature flags isn't obvious—I know that I certainly didn't "get it" at first. To help illustrate just how *wonderfully different* feature flags are, I'd like to step through the life-cycle of a single feature flag as it pertains to product development. This way, you can get a sense of how feature flags change your product development workflow; and, why this change unlocks a lot of value. For this thought experiment, let's assume that we're maintaining a collaborative task management product. Currently, users can create and complete tasks. But, they can only discuss these tasks offline. What we want to do is build a *simple* comment system such that each task can be backed by a persisted conversation. My goal here is *not* to outline a blueprint that *must follow* in your own projects. My goal is only to start shifting your mindset and your perspective on what is possible. ##### Flesh-out Work Tickets Traditionally, when building a feature locally, tasks in your ticketing system represent "work to be done". And, when using feature flags, this is also true. But, instead of arbitrary milestones, you need to start thinking about **each ticket** as a **deployment opportunity**. Not every ticket is going to represent a 1:1 relationship between code and deployment. But, starting off with this constraint will help you think about deconstructing the work into small meaningful changes. And, ordering those changes in such a way that deployments don't break production. Consider this list of tickets for our new commenting feature: * Configure feature flag. * Add link, route, and placeholder view. * Add new database table for comments. * Add command for creating a comment. * Flesh-out UI for adding a comment. * Add query for populating view. * Flesh-out UI for rendering comments. * Add command for deleting a comment. * Flesh-out UI for deleting a comment. * Add command for updating a comment. * Flesh-out UI for editing a comment. * Release feature. * **Remove feature flag.** When solving problems for our users, we want to focus on **getting to the User Experience (UX) as soon as possible**—a good user experience is ultimately what drives the success of a new feature. And, the sooner we can start iterating on the UX, the more opportunity we have to create a compelling experience. As such, when we consider both the breakdown and the ordering of tasks, we [usually] want to complete *just enough* back-end work to unblock front-end work. This breakdown may seem unnecessarily granular if you're used to building *entire features* locally. But, suspend your disbelief for a few minutes; and, I'll show you why taking small, incremental steps creates a lot of safety; and, opens the door to a more dynamic and reactive product development mindset. > **Caution**: The code samples in this chapter are going to be **heavily truncated**. I will do my best to keep them clear and insightful; but, please understand that there is much logic that I am omitting for the sake of brevity. > > ##### Configure Feature Flag As outlined in the previous chapter, this type of feature flag is what I consider to be a "Product" feature flag. Meaning, it is a temporary flag that only exists while our new feature is under development. To keep with my recommended naming convention, let's construct the feature flag using the `product-` prefix, the ticket number (of our Epic, User Story, Case, etc), and a descriptive suffix: `product-TASKS-103-comments` In most cases, a Boolean-based feature flag is the right choice. And, this case is no different. All we need to do here is hide the ingress to a new feature: a simple On/Off. Of course, since we don't actually have anything to show yet, let's configure our feature flag to be Off for *everyone*: ``` { "product-TASKS-103-comments": { variants: [ false, true ], distribution: [ 100, 0 ] } } ``` As you can see, 100% of all requests will be served the `false` variant for our new feature flag. ##### Add Link, Route, and Placeholder View Now that we have our feature flag configured, we can start building-out our feature. For the sake of simplicity, let's assume that we have a traditional, multi-page application (MPA); and, that our Task comments are going to be rendered on a new page within our application. The smallest, meaningful first step that we can take is adding the link that brings the user from the list of tasks over to the comments page (for a given task). This link is the ingress to our new feature; and, will need to be gated behind the feature flag. ``` <cfscript> canSeeComments = features.getVariant( "product-TASKS-103-comments", { key: request.user.id, userEmail: request.user.email } ); </cfscript> <cfoutput> <h1>Tasks</h1> <cfloop item="task" array="#tasks#"> <li> #encodeForHtml( task.description )# <a>Mark Complete</a> <cfif canSeeComments> <a>Comments</a> </cfif> </li> </cfloop> </cfoutput> ``` As you can see, we're evaluating the feature flag and using the returned variant to dynamically add HTML to the rendered output. Right now, the feature flag is configured to serve `false` for everyone; but, we know that we'll need to *conditionally enable* this feature in the near future. So, we're passing-in the `userEmail` for use in future targeting. Gating the *link* to our feature is an obvious requirement. But, less obvious is the fact that we need to gate the *destination page* as well. This serves both to block unauthorized access and to provide an "emergency shut-off" should something critical be exposed on the comments page. ``` <cfscript> canSeeComments = features.getVariant( "product-TASKS-103-comments", { key: request.user.id, userEmail: request.user.email } ); // Ensure that a user isn't attempting to bypass our // feature gate on the task-list page. if ( ! canSeeComments ) { throw( type = "FeatureNotEnabled", message = "Task comments not enabled for user." ); } </cfscript> <cfoutput> <h1>Task Comments</h1> <p>Coming soon...</p> <a>Back to Tasks</a> </cfoutput> ``` As you can see, this placeholder view does nothing more than check the feature flag state and provide a link back to the task list. At this point, we can ***ship it to production***! *Wait, WHAT?!* When you're new to feature flags, this first deployment can be *very uncomfortable*. It feels as though we're about to release something "broken". But, remember, the **magic of feature flags** is that we've *decoupled "release" from "deployment"*. We're not actually *"releasing"* any new functionality to our users—we're simply *"deploying"* code to the production servers (where it shall remain dormant until we enable the feature flag). And, once we confirm that *nothing looks different* in production, we can enable the feature flag for our *internal* users only. Once I have this basic gating in place, I start to differentiate between the **local environment** and the **production environment**. While the set of feature flags is the same across environments, the targeting within each feature flag is *unique* to the environment. To keep things simple, let's fully enable the feature flag in the development environment and only enable it for *internal* users in the production environment. ``` // LOCAL environment targeting. { "product-TASKS-103-comments": { variants: [ false, true ], distribution: [ 0, 100 ] } } // PRODUCTION environment targeting. { "product-TASKS-103-comments": { variants: [ false, true ], distribution: [ 100, 0 ], rule: { operator: "EndsWith", input: "userEmail", values: [ "@example.com" ], distribution: [ 0, 100 ] } } } ``` Locally, in our development environment, everyone gets the `true` variant making it easy to work on the new feature. But, in production, everyone still gets the `false` variant by *default*. Only users with an email address that ends in `@example.com` (our corporate email domain) will get the `true` variant and will be able to see the gated code. At this point, we can ***share the feature with our team***! *Wait, WHAT?!* If "shipping" incomplete code made you uncomfortable, "sharing" incomplete code with your teammates might make you downright anxious. We're now stepping into a radically different mindset: one of *inclusion* rather than *isolation*. Let's take a moment and consider the full journey of a product. First, it's nothing but an idea in your head. Then, perhaps you do a few rough sketches with pen on paper to help you mature those thoughts. And then, maybe you create a mock-up, which then gets turned into a prototype. And, eventually, code is written and deployed to production. At each one of those steps, the **fidelity** of your idea increases. And, with the increased fidelity comes a richer **feedback loop**. The link that we just added to our task list in *production* is going to *feel* different than it did in our *prototype*; which is going to *feel* different than it did in our mock-ups. The *sooner* we can get that live, production version back into the hands of our teammates, the *sooner* we can begin evolving the concept using real world evidence. > No plan survives first contact with the enemy. > > — Helmuth von Moltke the Elder > > ##### Add New Database Table for Comments Now that we have our placeholder view in production, we need to start fleshing-out the commenting experience. And, to do that, we need a way to store the comment data. Assuming that our application is already running on a relational database management system (like MySQL), let's create a new database table for our comments. At this point, the structure of our database table is going to be based on an *educated guess* as to how we believe the feature is going to work. We know that comments are going to be task-specific; and, that each comment will be posted by a user at a moment in time. With that information, we can create *something*. And at this point, we can ***deploy* that SQL table to production**. *Wait, WHAT?!* If shipping incomplete code made you uncomfortable, and sharing an incomplete feature made you anxious, deploying an "unused" database table might feel downright *backwards*. After all, isn't there a good chance that the structure of the database table will need to change once we start writing our code? Yes. And, that's OK. Remember, in a feature-flag-driven development workflow, I need you to think about each ticket as a *deployment opportunity* that, itself, represents a small, meaningful change to the system. Adding a new database table is meaningful in that it represents a persistence mechanism that now unblocks subsequent persistence coding. Our tickets no longer represent "completeness", they represent "progress." That said, databases *do feel different*, don't they—a bit "sticky" perhaps? Whereas code seems "fluid" and easy to change, database tables feel much more rigid—much more set in stone. We might even believe that database table structures have to be "correct" before we can deploy them. But, nothing in product development is ever "correct". Every aspect of our stack—from the code to the database to the platform—is nothing more than our *best effort* to solve problems with the information we have about the demands being placed on the system. As time passes, all of this changes; and, our system has to evolve with those changes. The platform evolves; the code evolves; and, so too must our database. The reality is, changing our database table structure is no different than changing our code. Yes, there are some additional complexities (such as those related to *rolling back* changes); but, the sooner we lean into the idea of a fluid, evolutionary database structure, the sooner we can start moving our product forward with ease and confidence. Now, just as a thought experiment, imagine that we get half-way through building this feature and we realize that there's something fundamentally wrong with our table structure and that we have to make a transformative change to it. Remember, customer aren't using it yet—it's still gated behind a feature flag. So, let's just `DROP TABLE` and start again. This is an extreme case. But, with feature flags gating access to our feature, even extreme cases like this can be met with comfort and without concern. ##### Add Command for Creating a Comment Now that we have our initial database structure in place, I like to think about the next *smallest, meaningful* step that we can take in building-out the commenting workflows. Creating a comment feels like a natural starting point. Though, "creating a comment" isn't just "one thing". As with many product interactions, there are both front-end and back-end aspects: the front-end, where the user initiates actions; and, the back-end, where said actions are translated into persisted state changes. If each ticket is meant to unblock subsequent work, we have to implement the server-side portion first. Then, once the server-side logic is ready, the relevant front-work can be started. The submission of a request (on the front-end) and the processing of a request (on the back-end) seem so deeply coupled that it can feel strange to work on them separately. But, while they do *work together*, they have *fundamentally different* concerns. Whereas the back-end is all about request routing, data validation, data normalization, state transformation, persistence, and error management, the front-end is all about information architecture, interaction design, affordance, usability, and communication. Working on only one-half of the interaction at a time allows you to be more focused in what you are trying to accomplish. This requires a smaller mental model which therefore incurs a smaller cognitive load. And, we'll talk about this more later in the context of team dynamics but, separating front-end work from back-end work also reduces code-review overhead and the mean time to approval. We've already gated the ingress to our task comments view behind a feature flag. And, depending on how we implement our routing, this may be sufficient. However, if we're exposing our "commands" through a set of API end-points, we'd want to gate that API access as well. For the sake of simplicity, let's assume that all of our task comment CRUD (Create, Read, Update, Delete) operations are being managed in a single resource. This makes it trivial to gate all operations behind a single feature flag check. Many web application frameworks have the concept of a `before()` method—a function that runs prior to any other resource method invocation. We can put our feature flag check there: ``` component { public void function before() { var canSeeComments = features.getVariant( "product-TASKS-103-comments", { key: request.user.id, userEmail: request.user.email } ); // Block any CRUD operations related to task comments // if the feature is not enabled. if ( ! canSeeComments ) { throw( type = "FeatureNotEnabled", message = "Task comments not enabled for user." ); } } // ... Rest of CRUD methods ... } ``` Not only does this `before()` method now gate our "create comment" command, it will naturally gate our future "delete" and "update" commands as well. This feature flag check is almost identical to the one at the top of our task comments view. And, seeing the *same code* in two different places can often be a "code smell" that indicates a need to refactor and abstract logic. There's certainly nothing wrong with moving both of these feature flag checks into a centralized location. However, it's important to remember that **all of this feature flag logic is temporary**. "Clean code" is important for long-term maintenance; but, over-thinking *intentionally temporary code* can be a needless waste of time. > **Permission**: When it comes to feature flags, copy-paste-modify is a perfectly acceptable technique. > > Once the ability to validate and persist a new comment is implemented, ship it to production! We might need to tweak it later once we start implementing the front-end logic; but, as is the ongoing theme of our work, that's OK. We *need to get comfortable* with the idea that our product is an *evolving* concept. ##### Flesh-out UI for Adding a Comment We've already gated access to the task comments page. We've already implemented the back-end logic for processing and persisting comment data. Now, we need to create the front-end HTML Form that allows a user to submit a new comment. Take a moment and just enjoy how narrowly focused our work is becoming? It's no longer a large, overwhelming set of requirements that you build over the course of *weeks or months*—each ticket now maps to a straightforward, well-bounded concept that you ship within *hours or days*. And, every time you complete a ticket and *ship code* to production, your entire team—from your boss to your project manager to your quality assurance (QA) engineers—gets a real sense of the progress you're making; and, how they can best set you and your project up for success. For example, the moment you ship this form to production, your QA team can start *fuzzing* the inputs, looking for edge-case you haven't considered. And, they can do this all in *parallel* with the work you're doing next. ##### Add Query for Populating View At this point, comments are getting created; but, they're disappearing into the ether. As such, the next meaningful change we can make is to start rendering those comments in the task comments view. And, just as with the creation of comments, we're going to break the rendering of comments into two separate tickets: back-end work and front-end work. You already know—more or less—how this page is going to work. So, don't fret over building the SQL queries before you have your UI (user interface). Lean into the idea that this can all change (as needed); and, do the smallest thing you can to keep the project moving forward. Ship those SQL queries to production! ##### Flesh-out UI for Rendering Comments The last thing we need to do in order round-out the creation of task comments is to render the comments to the screen. We already have our feature flag in place. We already have the data being pulled from the database. Now, it's just a matter of applying the right HTML, CSS, and JavaScript. And, here's the truly exciting part: once this UI is shipped to production, your team can start **using it**. And, I don't mean "testing it"—I mean *actually using it* in their day-to-day work. Can they delete comments yet? Nope. Can they edit comments yet? Nope. But, the moment your team can *create* and *view* comments, there's enough functionality in place to **fundamentally change the interaction model for the application**. And, the sooner you can get your team consuming this new functionality and the sooner you can gather meaningful feedback, the more likely you are to build a product that meets real world demand. ##### Add Command for Deleting a Comment There's nothing unique about deleting a comment. We're going to approach this in the same way we approached adding a comment: by building the back-end logic first and shipping it to production. ##### Flesh-out UI for Deleting a Comment And, once the back-end logic is in place, we can add the necessary user interface components for deleting a comment. And, of course, once that's done, we're shipping it to production! At this point, we have the ability to add a comment and we have the ability to delete a comment. Which allows us to start asking some uncomfortable questions. Like, **is this enough**? Is adding and deleting a comment a *sufficiently high* bar for this new feature? Consider other forms of media. For years, we couldn't edit or delete SMS text messages; and yet, people love sending texts. For years, we couldn't edit Tweets; and yet, people love posting on Twitter (now known as "X"). There is certainly precedent for success with a limited set of interactions. And, because feature flags allow us to build and deploy features *incrementally*, it means that we can start thinking much more critically about our product—and the value it adds—as it continues to unfold. **Our period of reflection is no longer limited by the anachronistic constraint of a fully-developed product**. ##### Add Command for Updating a Comment **[ON HOLD]**: As a company, we've decided that being able to add and delete comments is sufficient *for now*; and, that we need to stop and pivot to other higher-value-add work. ##### Flesh-out UI for Editing a Comment **[ON HOLD]**: See above. ##### Release Feature The magic of feature flags stems from our ability to decouple "release" from "deployment". We've already *deployed code* to production several times, but we haven't yet *released the feature*. Well, at least not to the "general audience". In fact, we released our feature to the *internal team* almost immediately after our first deployment. This allowed us to gather early feedback, tweak requirements, and ultimately **modify the scope of the project in real time**. And now that the `border-radius` on our buttons is smooth and our `box-shadow` game is on-point and our error messages provide helpful guidance, it's time to release our "task comments" feature to the world. No deployment is need—just a reconfiguration of our feature flag: ``` // LOCAL environment targeting. { "product-TASKS-103-comments": { variants: [ false, true ], distribution: [ 0, 100 ] } } // PRODUCTION environment targeting. { "product-TASKS-103-comments": { variants: [ false, true ], distribution: [ 0, 100 ] } } ``` The local development environment was already serving the `true` variant for all users; and now, so does our production environment. At this point, we need to start watching the error logs. Because, if there's any hard truth in this world, it's that users can always manage to break features in new and exciting ways! ##### Remove Feature Flag Once our feature has been *released* to the general audience, it's time to clean up the code. But, it's important to put *some amount of time* in between the release of the feature and the removal of the feature flag. This is to act a safety valve for any unexpected issues (such as a SQL query that suddenly start running slowly under increased load). > Everyone has a plan until they get punched in the mouth. > > — Mike Tyson > > When you first get started with feature flags, it can be tempting to leave feature flags in place for a long time (if not indefinitely). But, this is *fear-based thinking*. Feature flags add complexity to the control flow which, in turn, makes our code harder to comprehend and maintain. In some ways, they are a *necessary evil*. And, the moment you can remove a feature flag, you should. This is where a good naming convention comes into play. We named our feature flag `product-TASKS-103-comments`. This makes the relevant gating logic *extremely easy* to find in our codebase. There's no chance that this text will be used in any other context. But, imagine if we had named the feature flag something simple like, `comments`. How confident would you feel about finding and removing all the relevant code? ##### Celebrate In building this new feature, we did some scary things! We shipped code early; we shared the feature early; and, we started consuming our own work far before it was complete. And, in doing so—by leaning into that discomfort—we were able to parallelize work across the team, adjust the product in response to early feedback, and, ultimately, re-shape the roadmap even as it was being developed. Are you beginning to see the possibilities? #### [#](https://featureflagsbook.com/#UseCases)Use Cases When your feature flags implementation can store JSON variants, it means that there's no *meaningful limitation* as to what kind of "state" you can represent. Which means, the use cases for feature flags are somewhat unlimited as well. As we saw in the previous chapter, using Boolean-type flags to safely build and then incrementally release a feature is going to be your *primary gesture*; but, the area of opportunity will continue to expand alongside your experience. And, to help jump-start your imagination, I'd like to touch upon some of my use cases. This is not intended to be an exhaustive list, only a list of techniques with which I have hands-on experience. ##### Product: Feature Development - General Audience Building new features *safely* and *incrementally* for your general audience is the bread-and-butter of feature flag use cases. This entails putting a new feature behind a feature flag, building the feature up in the background, and then incrementally rolling it out to *all customers* within your product. We've already seen this in the previous chapters; so, I won't go into any more detail here. This type of product feature flag is intended to be removed from the application once the feature has been fully released. ##### Product: Feature Development - T100 At work, the "T100" cohort refers to our top 100 customers in terms of both revenue and potential expansion. In an ideal world, new features get used by *all customers*; but, in a pragmatic world, sometimes you *just have to* build a feature that only makes sense for a handful of high-priority customers. To do this, you need to identify a property that allows for an elastic cohort of users. For example, you might use an email address suffix for "Acme, Inc.": Remaining chapter content available in purchased book (coming soon). #### [#](https://featureflagsbook.com/#BridgingtheSophisticationGap)Bridging the Sophistication Gap As you're reading through the aforementioned use cases, it might occur to you that large, sophisticated companies have different ways of solving the same problems. For example, instead of using feature flags to implement rate limiting, a more sophisticated company might put that logic in an Application Load Balancer (ALB) or a reverse proxy. Or, instead of using feature flags to implement IP-blocking, a more sophisticated company might add reactive request filtering to a Web Application Firewall (WAF) or dynamically update ingress routing rules. Or, instead of gating code behind a feature flag, a more sophisticated company might adjust the traffic distribution across an array of Blue/Green deployment environments. Or, instead of using feature flags to adjust log emissions, a more sophisticated company might change the filtering in their centralized log aggregation pipeline. Large, sophisticated companies can do large, sophisticated things because they have massive budgets and hordes of highly-specialized engineers that are focused on building specialized systems. This allows them to solve problems at a scale that most of us cannot fathom. But, this difference in relative sophistication isn't a *slight* against feature flags. Exactly the opposite! The "difference" is a *spotlight* that underscores the outsize value of feature flags—that we can use such *simple and straightforward* techniques to solve the same class of problems (**at a fraction of the cost and complexity**). Nothing illustrates this as clearly as **load testing**. When a sophisticated company needs to load test a new feature—that is, to test whether or not the system can handle a large volume of requests in parallel—it sets up a load testing environment. In order for a load test to be meaningful, the "test environment" has to mirror the "production environment" as closely as possible. Which means: * The size and configuration of each underlying test server has to match production (in terms of CPU, RAM, and IOPs allocation). * The horizontal scaling of these test servers has to match production so that they can dynamically spin-up in order to accept increasing load. * The size and configuration of the underlying databases has to match production (in terms of CPU, RAM, and IOPs allocation) so that they can use the same amount of "working memory" without having to swap on disk. * The data volume and data diversity within the test databases has to match production so that the table statistics lead to the same query plan executions. But, of course, this data needs to be sanitized for security and compliance reasons; which means, there's probably some scheduled task that is continually keeping the test databases synchronized—and anonymized—with production. * There needs to be some sort of "traffic replay" that is putting load on the test servers that is unrelated to the feature being tested. This way, the test servers aren't *just* testing the new feature - they're testing the new feature *on top of* the normal server traffic. * And, of course, there needs to be something performing the actual load testing of the new feature itself. As you can imagine, setting up and consuming a test environment like this is **very complicated** and **very expensive** (both in time and in money). And, the ugly truth is, no matter how many resources get poured into a testing environment like this, it will forever be a "best approximation" of what production is *actually* like. Because, **there's no place like production**. Now, you may not have noticed, but we already talked about "load testing" in a previous chapter; only, we called it an "incremental roll-out". Consider for a moment *why* sophisticated companies jump through so many expensive hoops in order to set up a testing environment: **safety**. The *isolation* provided by the testing environment means that, no matter what happens during a load test, the production environment will remain unaffected. This is *exactly* what an *incremental roll-out* does. Only, without the exorbitant cost and complexity. When we gate a feature behind a feature flag, the incremental roll-out of said feature allows us to see if-and-how it might negatively affect production—well before any harm is actually inflicted. A *vanishingly small* number of changes to an application actually merit load testing. But, for the sake of discussion, let's say that we're about to release one such change. We can start off by releasing the change to *our user only* (with ID: `123`): ``` { "really-scary-change": { variants: [ false, true ], distribution: [ 100, 0 ], rule: { operator: "IsOneOf", input: "userID", values: [ 123 ], distribution: [ 0, 100 ] } } } ``` With this change, every single user in the application will continue to receive the traditional experience while we—a lone engineer—can go through the application and try out the new experience. There's *almost* nothing that a *single user* can do to take down production. But, starting with such a narrow release can certainly give us an early indication of both qualitative issues (ie, does the new workflow *feel* slow) and quantitative issues (ie, are there any metrics or dashboards that now show an irregularity). This narrow testing can last a few minutes; or, it can last a few weeks. It doesn't much matter because there's *zero cost* associated with your level of caution. No test servers are running. No test databases are running. It's just the production servers and some internal control flow branching. And, if all signs look good, you can *slowly* widen the release. Perhaps moving onto your internal team (with email domain: `@example.com`): ``` { "really-scary-change": { variants: [ false, true ], distribution: [ 100, 0 ], rule: { operator: "EndsWith", input: "userEmail", values: [ "@example.com" ], distribution: [ 0, 100 ] } } } ``` Continue checking the errors logs and the dashboards to see how the system is responding. Are there qualitative issues? Are there quantitative issues? And, if all signs look good, you can *slowly* widen the release. Perhaps to 1% of the general audience: ``` { "really-scary-change": { variants: [ false, true ], distribution: [ 99, 1 ], rule: { operator: "EndsWith", input: "userEmail", values: [ "@example.com" ], distribution: [ 0, 100 ] } } } ``` And then 2%. And 3%. And so on. All while watching the system for contraindications. At each increment, you can remain confident that what you're seeing is going to be reflective of the production servers because this **is the production environment**. Congratulations, you just ran a load test safely in production! Now, I'm not saying that feature flags *completely replace* all of the sophisticated techniques that sophisticated companies use. But, I am saying that, with feature flags, relatively simple companies can reap *many of the same rewards* with only a *fraction of the cost and complexity*. #### [#](https://featureflagsbook.com/#LifeWithoutAutomatedTesting)Life Without Automated Testing > People who say it cannot be done should not interrupt those who are doing it. > > — George Bernard Shaw (misquoted) > > I have a dirty little secret: **I haven't written a test in *years***. And, I deploy code *to production* anywhere between 5 and 10 times a day. Now, I'll caveat this by saying that I don't write and distribute "library" code for broad consumption—I maintain a large SaaS product over which the company has full end-to-end control. If your context is different—if the distance between your code and your consumer is vast—this chapter may not apply to you. To be clear, I enthusiastically support testing! In fact, I *thoroughly test* every single change that I make to the code. Only, instead of maintaining a large suite of automated tests that run against the *entire codebase*, I *manually test* the code that I change (right after I change it). There is a common misconception in the engineering world that *automated* tests and *manual* tests are at opposite ends of the testing spectrum. Some engineers might even go so far as to suggest that automated testing is *intended to replace* manual testing. But, manual testing *isn't optional*. At least, not in the "product" world. Manual testing is an essential part of the product development life-cycle. You make a change, **you try it out**, you make sure it works, **you see how it feels**, you adjust the code as needed. Building a successful product isn't *only* about "correctness"—it's about creating an "experience". And, you'll never create a great product unless ***experiencing the product*** is a core—*and consistent*—part of your product development workflow. > What's true of all bugs seen in production? They passed all the tests. > > — Rich Hickey (creator of Clojure) > > Now, I'm not here to tell you that manual testing *replaces* automated testing. Because—again—this would be a misunderstanding of the role that various testing methodologies play. But, I will say that manual testing is *mandatory* and automated testing is *optional*. And, as with all things "optional", it becomes a matter of investment: is the cost of adding and updating automated tests worth the value that they deliver? If your answer is an emphatic, *"Yes"*, then, by all means, please continue writing your tests. But, let's step back and examine why automated tests *can be* a value-add: they instill **confidence**. The confidence that you're not about to make a mistake and break production in an unexpected way. Or, to frame it differently, it's precisely your *lack of confidence* in the code that acts as the reason to write automated tests. After all, if you had *great confidence* that the code you wrote is correct, there'd be no justification for spending even more time / money / opportunity costs to add *yet another* layer of confidence. Feature flags create that type confidence. In fact, I would argue that feature flags create *more confidence* than automated tests. By their nature, automated tests can only test the "known knowns" within an application. And, for that matter, they only cover the *subset* of known knowns that the developer *thought to consider* in their tests. Feature flags, on the other hand, allow us to react to *anything*—both the known knowns and the unknown unknowns. When you deploy something *behind* a feature flag; and, when enabling that feature causes an error; you just *disable the feature flag*. It doesn't matter *why* the error occurred—if it was a predictable issue or a novel edge-case—it only matters that we can react to the issue quickly and return production to a stable, working condition. Of course, this confidence won't manifest in isolation. It is a byproduct of the way in which **feature flags transform product development**. Consider the list of tickets that we closed in an earlier thought-experiment (see **[Life-Cycle of a Feature Flag](https://featureflagsbook.com/#LifeCycleofaFeatureFlag)**): * Add link, route, and placeholder view. * Add new database table for comments. * Add command for creating a comment. * Flesh-out UI for adding a comment. * Add query for populating view. * Flesh-out UI for rendering comments. * Add command for deleting a comment. * Flesh-out UI for deleting a comment. * Add command for updating a comment. * Flesh-out UI for editing a comment. * Release feature. If I had completed all of these tickets *locally*; and, then waited until the feature was *completely done* before deploying it; and, if deploying it meant releasing the feature to *all users* at the *same time*—yeah, **I'd be terrified**! When working under these constraints, I'd be more shocked if I *didn't break production*. I *used to* work this way. My team *used to* work this way. And—in fact—we broke production so many times that I would often talk about "taking down production" as being a "rite of passage" (see **[Of Outages And Incidents](https://featureflagsbook.com/OfOutagesAndIncidents)**). But, incidents went from happening several times of day to being a rare occurrence. We didn't make this change by *adding lots of tests*—we did it by *adding lots of feature flags*. And, the more we baked feature flags into our workflow, the more stable everything became. It turns out, *confidence* has an inverse relationship to the *breadth* of work being done. Change a *lot of things*—as we discussed above—and confidence plummets. Change *nothing*, and you can be infinitely confident that you broke *nothing*. Feature flags allow us to get as close as we can to the "nothing" end of that spectrum while still moving the application forward. By decomposing large efforts into a series of small, independently deployable steps, it keeps the breadth of individual changes very low; which, in turn, keeps our degree of confidence very high. Small changes also mean small PRs (Pull Requests). And, as we'll cover in a future chapter, small PRs beget better reviews. And, better reviews mean that fewer issues ever make it into production. But, perhaps most importantly, feature flags slowly change the way that we approach application architecture itself. Since feature flags always require code to be *added (safely)* and then later *removed (safely)*, we're forced to start writing code that is: 1. Easy to build in isolation. 2. Easy to find. 3. Easy to delete. These three properties *necessarily* end up creating code that is **loosely coupled** to the rest of the application. And, this loose coupling ends up being one of the most transformative changes when it comes to writing code with confidence. Consider what it is that engineers are *really* afraid of when they write a lot of tests. It *usually isn't* the obvious knock-on effects—it's the quantum entanglement; the, "spooky action at a distance." They are afraid that when they change this line of code *"over here"*, something *"waaaay over there"* breaks in an unexpected and non-obvious way. When you write code using feature flags; and, when you organize code so that it's easy to isolate, find, and delete; this entanglement—this tight coupling—it stops happening. And, when the tight coupling goes away, so too will *many* bugs. Manual testing then becomes a "commensurate" activity. Meaning, you test what you changed and no more. If I bring up the idea of manual testing to people who love automated testing, they will immediately object: > But, then you have to regression test the entire application every time you make any change. > > No—this is incorrect. Or, I should say, this is *only correct* when the degree of coupling is *so high* that the blast radius of your changes cannot be predicted. Working in small, incremental steps removes this FUD (Fear, Uncertainty, Doubt); and, makes the outcome of each individual change **small, safe, and obvious**. Over time, using feature flags simply forces your code to become easier to maintain. And, this isn't just for *you*, it's for *everyone*. All of the rewards that you get from feature flags are going to be the same for the next developer; and, all the developers after that. Because the safety is *baked into the process itself* (and, into the architectural patterns that feature flags encourage). > **Aside**: After maintaining a single product for over a decade, one thing that I've come to understand is that there is a lot of code in your application that *never changes*. A beautiful side-effect of manual testing is that you never have to worry about testing the code that never changes. > > There are, of course, limits to what feature flags can do. Not all changes fit neatly into a feature flag workflow. Which means, some of the *broader changes*—like updating the version of your front-end framework—will require more extensive regression testing. But, this is a relatively rare occurrence. And, it's an occasional cost that I'm very happy to pay. In the end, no approach is perfect. Everything is a calculated trade-off. Automated testing *does* have some advantages. But, feature flags completely changed the product calculus; and now, my team is able to maximize the time that we spend *building value-add features* for our customers. And, we do so with confidence. #### [#](https://featureflagsbook.com/#CodingDefensively)Coding Defensively * Designing for failure. * Designing with EOL in mind. * Rolling-deploys. * Rolling back code. * Long-lived processes. #### [#](https://featureflagsbook.com/#ServerSidevsClientSide)Server-Side vs. Client-Side * Coping with different rates of change. * Keeping complexity low. * No "prop drilling". #### [#](https://featureflagsbook.com/#KISSKeepItSuperSimple)KISS: Keep It Super Simple * Tips of lowering complexity. * Targeting users. #### [#](https://featureflagsbook.com/#OwnershipBoundaries)Ownership Boundaries Feature flags are an **implementation detail** for engineering teams. As such, **each feature flag belongs to the engineering team that created it**. If we were talking about any other low-level algorithmic decision, this sense of ownership would be self-evident; and, we wouldn't need to have this discussion. However, since feature flags can be both observed and configured *outside* of the application itself, it's easy for people to *misunderstand* where the boundaries need to be drawn. ##### Who Can Manage a Feature Flag Simple—the engineering team that *implemented* the feature flag is the only team that can *manage* the feature flag. Since feature flags are an *implementation detail*, only the *implementing engineers* have the necessary understanding of how changes to the configuration will impact the production system. These engineers know which database queries are involved; which API calls are being made; which areas of the application are being touched; and—*most importantly*—which error logs and performance dashboards need to be monitored during the roll-out. Your product manager doesn't have this information. Your designer doesn't have this information. Your data scientist doesn't have this information. Your growth engineers don't have this information. Your security and compliance officers don't have this information. No one other than the implementing engineers have a fully integrated understanding of *the implications*; and, therefore, no one other the implementing engineers should be touching the feature flag. At the beginning of this book, we talked about feature flags as being little more than dynamic `if` statements. This code-oriented perspective can help bring the ownership boundaries into focus. The people who have permission to configure a feature flag are the same people that have permission to open your code and edit your `if` statements. And, if you wouldn't be comfortable with someone jumping in and editing *your code*, you *shouldn't be* comfortable with that person jumping in and editing *your feature flag* configuration. The exception to this rule involves open-ended, longer-term feature flags. For example, if a feature flag is being used to gate T100 features or to act as a make-shift paywall (see **[Use Cases](https://featureflagsbook.com/#UseCases)**), targeting of the feature flag can be safely managed by customer-facing employees. Remaining chapter content available in purchased book (coming soon). #### [#](https://featureflagsbook.com/#LogsMetricsandFeatureFlags)Logs, Metrics, and Feature Flags * Table-stakes for robust application development. #### [#](https://featureflagsbook.com/#TheHiddenCostofFeatureFlags)The Hidden Cost of Feature Flags > Programmers know the benefits of everything and the trade-offs of nothing. > > — Rich Hickey (creator of Clojure) > > Feature flags are an essential part of modern product development. But, every architectural choice represents a trade-off; and, so do feature flags. While they *do* create a lot of value, that value comes at a cost—both literal and figurative. Whether you're building your own feature flags implementation or you're using a 3rd-party vendor, feature flags cost money—actual dollars-and-cents. You're either paying for development time and hosting (if you build your own implementation); or, you're paying for usage and seats (if you buy a managed solution). But, we'll talk more about the financial trade-offs later (see **[Build vs. Buy](https://featureflagsbook.com/#BuildvsBuy)**). For now, I want to focus on the more hidden cost of feature flags. In terms of developer ergonomics, feature flags create additional complexity in your code. By definition, they are spreading the business logic across two different systems: the application code and the runtime configuration. This makes it much more difficult to understand why the application is exhibiting a certain behavior. And, there's *no possible way* to know—at a glance—which conditional branch in the code should be executing. Combine this with the fact that feature flag configurations are unique across environments and it's completely understandable as to why people *fall down* accidental rabbit holes. I can't tell you how many times I've seen a good developer lose *multiple hours* of productive time, not realizing that a feature flag was *enabled in production*, but *disabled* in the local development environment (or vice versa). Feature flags clearly reduce cognitive load by allowing large tasks to be atomized. But, from a workflow perspective, feature flags incur a higher cognitive cost. When you build a product using feature flags, you are *forced* to think more deeply about the process of development. And, once the development is done, you are *required* to be much more diligent about your clean-up. Of course, I say "required" here in the *moral sense*, not in the business sense. Because, at the end of the day, no one is going to step in and actually force you to clean-up after yourself. In fact, your manager(s) may actively encourage you to skip the clean-up process so that you can move onto other "higher priority" items. Embracing the importance of clean-up and then advocating for—and, perhaps, even fighting for—such work is uncomfortable. And, it will likely necessitate a higher degree of grit and conviction than you are accustomed to. Because, after all, the system "works" even when you leave the "dead code" in place. But, death does not exist in isolation. And as the dead code sits there and "rots", sepsis begins to set in. Over time, the application becomes increasingly complex and ambiguous. In turn, developers become ever more afraid of deleting code because they are increasingly unsure as to what the code actually does. This, in turn, compounds the overall complexity problem, slowly and steadily grinding the application to a halt and undoing all of the advantages that feature flags gave us in the first place. It is on your shoulders, as the developer, to keep the application code in a healthy, predictable state. Which means, you must wake up every day and *recommit* to doing the *right* thing. Over and over again. This includes: * Creating tickets to remove feature flags so that you don't forget—even if you know that these ticket won't be relevant for weeks (or even months) in the future. * Actually executing on said tickets—even if management doesn't think it's worth the effort. * Negotiating with other teams to enforce a finite timeline for feature flags. * Removing all outdated control flow logic. * Removing any *deeper* dead code that is no longer referenced by the outdated control flow logic. * Performing the previous step, recursively, until no more deletions are possible. * Removing feature flag definitions from the administrative system. * Doing all of this for other developers who were too lazy to do it for themselves (this is a sad, but inevitable, truth of collaborating on a system with other people). It takes dedication, often times in the face of frustration. But, I promise you that the value-add is worth the price of admission. #### [#](https://featureflagsbook.com/#NotEverythingCanBeFeatureFlagged)Not Everything Can Be Feature Flagged Once you begin weaving feature flags into your product development life-cycle, it can become hard to imagine any other way of working. But, not every type of change can—or should—be put behind a feature flag. Sometimes, the level of effort just isn't worth the added complexity. At the end of the day, *anything is possible* when given the necessary time and resources. But, not everything is worth the cost of implementation. **Feature flags should be *relatively easy* to use**—they should be simplifying the amount of work you have to do in order to make changes in a production application. If the path forward isn't clear; it's likely that the change you're making isn't a good match for feature flags. And, that's OK. Feature flags are a tool. You use them when they make sense. And, you choose another tool when they don't make sense. In my experience, feature flags are most effective for *localized changes*. That is, when the changes live within a relatively small boundary. Feature flags are much *less effective* when the changes have broad implications or require many files to change simultaneously. Some examples of work that doesn't play well with feature flags: * **Updating a database version**. Not only does updating a database require downtime for your application, version changes can have knock-on effects for *every single query* in your application. Attempting to branch the logic of *every single query* is an extraordinary cost that's likely to create bugs, not prevent them. * **Updating a JavaScript framework**. If you're updating a JavaScript framework *without* making any other front-end changes, you might possibly gate the `<script>` tag behind a feature flag. But, if your JavaScript framework has any *breaking changes* within the update, those breaking changes can have knock-on effects that live anywhere in the application. Trying to gate both the *version* of your framework and the affected front-end code is going to add massive complexity. * **Updating a CSS framework**. CSS doesn't exist in a vacuum. It works in conjunction with HTML (and can often act as a "hook" for JavaScript DOM queries). Which means, when you change the CSS in a large refactoring, you often have to change a lot of your HTML structure as well. Trying to gate both your `<link>` tag and the affected front-end code is going to add massive complexity. * **Updating your application runtime**. Feature flags operate within your application runtime. As such, they can't be used to determine which application runtime is currently operating. Are you seeing the common thread? Any change that *touches* a lot of files or has *implications* for many files is not worth the effort (in terms of feature flags). And that's if it's even possible at all. Feature flags are amazing; but, they are not a panacea. For this type of work, you'll have to continue using whatever strategies you've been using historically. #### [#](https://featureflagsbook.com/#BuildvsBuy)Build vs. Buy As engineers, we *love* to build! As such, it's completely naturally to learn about feature flags and then think, *"Yeah, I could build that."* And, in fact, we did build a *very simple* implementation earlier in this book (see **[Going Deep on Feature Flag Targeting](https://featureflagsbook.com/#GoingDeeponFeatureFlagTargeting)**). But, building your own feature flags system is *probably* not a great use of your time (or your company's time). Feature flags are **deceptively simple**. So much of what we see—the branching logic within our application code—is just the tip of the iceberg. There's much more below the surface that goes into making your feature flags available, reflecting changes in real time, and allowing your team to effortlessly manage feature flag state across multiple environments. Assume that the *cost* of building your own implementation is *going to be high* (higher than you anticipate). And then, ask yourself if this kind of work generates a competitive difference for your company? Meaning, does *building* your own implementation give you a competitive advantage when compared to *buying* an existing implementation? The answer is *probably*, No; but, it's a conversation worth having internally. After all, feature flag services are *not cheap*. And, the more your team grows, the more expensive these offerings become (based on a per-seat pricing model). Each SaaS vendor is going to be different; but, consider this list of common features: * Project management - a bounded collection of feature flags. * Environment management - a bounded set of targeting rules. * Feature flag management - defining variants and targeting rules. * User management and permissions. * Insight and analytics - which variants are being expressed in your application runtime. * Cohort management and segmentation - defining reusable targeting rules. * Change management and approvals - tracking and gating changes to feature flag state. * Software Development Kits (SDKs) - libraries already authored (and battle-tested) for all of your programming languages. * Code insights - finding and removing feature flags in your application. It's a matter of ROI (Return on Investment). If you *buy* a solution, is the value it delivers worth the **explicit cost** you're paying? And, if you *build* a solution, is the value it creates worth the **opportunity cost** that you're giving up—the time you *could have spent* building a differentiating feature for your customers? #### [#](https://featureflagsbook.com/#TrackActionsNotFeatureFlagState)Track Actions, Not Feature Flag State > If you graph the numbers of any system, patterns emerge. Therefore, there are patterns everywhere... > > — Maximillian Cohen ([Pi, 1998](https://www.imdb.com/title/tt0138704/)) > > Data Scientists and Growth Teams love to track everything. The assumption being that with enough data, user behaviors can be predicated and then manipulated. As such, when feature flags are introduced to an application, some people will want *feature flag state* to be included in the existing analytics data. In my experience, this is a bad idea. For starters, it adds *even more* complexity to the code, especially if tracking happens in both the front-end and the back-end. But, more than that, tracking feature flag state cuts *against the grain* of a feature flag's expected life-cycle. Remaining chapter content available in purchased book (coming soon). #### [#](https://featureflagsbook.com/#ACulturalRevolution)A Cultural Revolution At this point, I hope that you can see how feature flags make engineering work safer and more effective. In fact, I'm hoping that you're **downright eager** to start using feature flags as soon as possible! And, I'm confident that once you start, your whole perspective on product development will be forever changed. But, the "technical" benefits of feature flags are *only the beginning*. If you work on a team with other engineers, product managers, and designers, there's much more value to be had. The rest of this book deals with the inter-personal aspects of using feature flags. And, I do *not* exaggerate when I say that feature flags *can* lead to a full-on **cultural revolution** within your company. Are you ready to take the red pill and see just how deep the rabbit hole goes? #### [#](https://featureflagsbook.com/#PeopleLikeUsDoThingsLikeThis)People Like Us Do Things Like This * On the culture of shipping and serving customers. * Culture is created from the bottom-up. #### [#](https://featureflagsbook.com/#TheDemocratizationofProductDesign)The Democratization of Product Design * Roles are helpful, but they can also be a burden. * Our deepest fear. * Change requires tension. * Asking for forgiveness, not permission. #### [#](https://featureflagsbook.com/#CocreatingtheMVPMinimumViableProduct)Co-creating the MVP (Minimum Viable Product) * By working incrementally, we bring everyone to the table. * This includes customers. * Fallacy of Henry Ford. * Incremental fidelity, from paper sketching to production. #### [#](https://featureflagsbook.com/#AnOpinionatedGuideToPullRequestsPRs)An Opinionated Guide To Pull-Requests (PRs) * A culture of shipping doesn't happen by accident. * Shipping requires intention from the start of code to the deployment of code. People have to buy into the process. * Code completed is more important than code being written. #### [#](https://featureflagsbook.com/#DynamicCodeDynamicTeams)Dynamic Code, Dynamic Teams * When value created means value delivered, we don't fall victim to sunk cost fallacy. #### [#](https://featureflagsbook.com/#RemovingtheCostofContextSwitching)Removing the Cost of Context Switching * Small goals mean less cognitive load. #### [#](https://featureflagsbook.com/#MeasuringTeamProductivity)Measuring Team Productivity * Goodhart's Law: When a measure becomes a target, it ceases to be a good measure. * When we create a culture of shipping, the act of shipping becomes a meaningful metric. * Fuzzy math over time still shows trends. #### [#](https://featureflagsbook.com/#TheGoalGradient)The Goal Gradient * The psychological challenge of long journeys., From Concept to Cultural Revolution, featureflagsbook.com, 01hjnwmp1pkfhzayt5madmsdp8.