Why You Can't Build AI Without Progressive Delivery  ## Document Notes ## Highlights - **Progressive Delivery As Engineering Physics** - Progressive delivery measures and manages the "jerks" in software delivery using the four A's: abundance, autonomy, alignment, and automation. - Treat these forces like physics to reduce surprising behavior and improve user outcomes. Transcript: Alex Williams Of the concepts of physics that apply to progressive delivery. And this is the beginning of, you know, where you present the main themes of the discussion, which I thought we would talk about, the four A's. And that's the framework that you use for progressive delivery. And you write, the evolution of progressive delivery has been shaped by technological advances, much as physics has evolved to measure and manage forces of motion. I thought that was pretty cool. James Governor Well, that's Adam Zimman. I'm not taking any credit for that. So Adam Zimman, he's amazing. Really enjoyed. So Adam, progressive delivery, I think any, and it's interesting because the term has been fairly widely adopted. People do use term now. It's always amazing when somebody you didn't know or you hear a presentation at an event or something and someone talks about progressive delivery and you're like, oh, people actually Do use this term. But the thing about... Alex Williams It's, you know, no pun intended, but it's a flag you can mark in your career. James Governor Oh, there you go. So the thing with Adam that was interesting to me was I had been thinking, basically the whole idea began with rage and anger. Oh. Yeah, yeah, no. I mean, it was, you know, a good developer will occasionally rage code something. Yes. You know, something will annoy them so much they write their own. Right. For me, I had been to KubeCon and I was so annoyed at the failure to provide any use cases for service mesh. They kept just saying Istio is good. I had this bee in my bonnet about how dumb it was that they were unable to talk about what you might be able to use that for. And a couple of weeks later, I go and I meet a guy from Microsoft, Sam Kuchenheimer, and he was talking about something that Microsoft had done back in the day, progressive experimentation. And what I realized that there were a whole bunch of things that we needed to bring together, really thinking about, again, putting the user first, being able to roll things out to a specific Cohort of users before we rolled it out more broadly to have better experiences. I was like, you know what? And to my mind, at least, some of the automation capabilities with service mesh, I was like, okay, here at least is a use case for this. And you have an idea like that at the at that point you're just someone dance you know they all meme someone's dancing crazy on a hill just they look like an idiot and then then the second Person that comes and dances with them on the hill right and then suddenly everyone joins everyone's dancing so adam zinman was the second person that made me not a crazy person on the Hill. And he came along. And yeah, the physics stuff, that was something he wanted to add to the book. Alex Williams Yeah. And so let me just give an intro here to the discussion about the four A's because there's this concept of jerk. It's like when you're on a subway and you're holding the pole and the train kind of stops and jerks. That's the jerk, right? That's the physics. That's the motion, that happens. You're trying to eliminate that as much as possible. And so the goal is to help us measure and manage the technological jerks in our system, the four A's, abundance, autonomy, alignment, and automation. And then you go through, like, all these things that have led up to that and everything else. So how did you come up with these forays? So like ([Time 0:04:25](https://share.snipd.com/snip/5031172c-bf65-4578-9de7-9c8b5fe50c3c)) - **Idea Born From KubeCon Frustration** - James Governor describes his early inspiration for progressive delivery as born from frustration at KubeCon and lack of concrete use cases for service mesh. - Adam Zinman later validated and joined the idea, turning it into a broader movement. Transcript: James Governor Oh. Yeah, yeah, no. I mean, it was, you know, a good developer will occasionally rage code something. Yes. You know, something will annoy them so much they write their own. Right. For me, I had been to KubeCon and I was so annoyed at the failure to provide any use cases for service mesh. They kept just saying Istio is good. I had this bee in my bonnet about how dumb it was that they were unable to talk about what you might be able to use that for. And a couple of weeks later, I go and I meet a guy from Microsoft, Sam Kuchenheimer, and he was talking about something that Microsoft had done back in the day, progressive experimentation. And what I realized that there were a whole bunch of things that we needed to bring together, really thinking about, again, putting the user first, being able to roll things out to a specific Cohort of users before we rolled it out more broadly to have better experiences. I was like, you know what? And to my mind, at least, some of the automation capabilities with service mesh, I was like, okay, here at least is a use case for this. And you have an idea like that at the at that point you're just someone dance you know they all meme someone's dancing crazy on a hill just they look like ([Time 0:05:41](https://share.snipd.com/snip/641fff34-e14c-4dc9-a3ff-6cc4f73abf91)) - **Cloud Enables Experimental Abundance** - Cloud and open source created abundance that enables experiments and multiple deployment environments at low cost. - This abundance changes what delivery practices like blue-green can reasonably assume and enables more experiments. Transcript: James Governor Yeah. Well, I think from our perspective, one of the things about, you know, you're going to come out and talk about, you know, something like progressive delivery. You're immediately like, wait a minute, James. You know, what are you smoking here? Because, you know, we've got continuous delivery and we've got this amazing work that the industry has done that, you know, individuals, you know, I mentioned Jean, but, you know, You know, we could talk about, you know, like Jess Humble. You know, I think if we think about DevOps, we could talk about like Nicole Foolsgren, the amazing work that she's done in terms of measuring better outcomes for software delivery. If you're going to come out there and say progressive delivery, then you're going to have to think about what are we actually talking about? So if we think about CICD, those concepts, they were pre-cloud. And it really struck me when I was thinking about what we do differently today, that the cloud provides abundance in a way that enables us to do things we just could not otherwise have Done. So if we're going to do, you know, we talk about blue-green deployments. You literally originally had to have two completely separate pieces of infrastructure, you know, sets of servers, storage, application, networking. You'd have two of those and do a cutover. Obviously, with virtualization, you know, that began to become, break down some of those barriers. Right. But the idea of you would actually need to have two environments. What if you wanted to do three or four? With the cloud, you can spin up as many instances as you want. And that abundance was not, didn't exist in the world when some of these key concepts about building better software came out. So I was like, what was different when they wrote the book on continuous delivery? The cloud wasn't there. So abundance, I was really talking, and again, open source has completely changed our ability. If you had to constantly give Oracle every time you wanted to do anything with the database, you couldn't be, it's so much harder to experiment, to run experiments, to do. So yeah, Abundance, it was like, that's the difference, is the cloud, open source, social, have utterly changed our capability to collaborate and build software. And so, yeah, that was that was where abundance came from. ([Time 0:08:00](https://share.snipd.com/snip/d06adc3b-e153-43e1-9d90-35fea844f898)) - **Use Feature Flags For Controlled Rollouts** - Use feature flags and platform tooling to run targeted experiments and manage rollouts at scale. - Enable teams to test cohorts before broad releases to improve user experience. Transcript: James Governor And so, yeah, certainly Edith really understood, I think, how software delivery was changing. And yeah, I think that feature flags, feature management, absolutely key in being able to do something like progressive delivery. Alex Williams So the ([Time 0:13:38](https://share.snipd.com/snip/972433de-1810-4ec0-a07e-b4b5fe364d47)) - **Autonomy Reduces Delivery Bottlenecks** - Autonomy helps developers and small teams deliver faster by reducing permission bottlenecks. - Granting responsibility and freedom reduces delivery "jerks" and improves outcomes. Transcript: Alex Williams So the second A in the four A's is autonomy. And it's described in one sentence. Autonomy is the ability of an individual to act independently from others. James Governor Yeah, 100%. Now look, software development is a team sport. Absolutely. But one of the things that we've seen, again, and that is different in how the industry is working is, again, I discussed the developer not needing to ask for permission so much. Developers do their best work when they have autonomy, right? Humans do their best work when they have autonomy, when they're allowed to get on with the job at hand. And so if we think about what we delegate in terms of responsibility and who is doing the work and making sure they have the autonomy to solve the problem, making sure that they're in a Position to, as I say, understand a user experience, that's one of the things that I think as an industry, we've really moved ahead in terms of, yeah, absolutely, the ability of, again, Look at Amazon, the two-pizza team, right? It is a small, autonomous team getting on with its work. And I think those are some of the, you know, that's certainly not how it worked in the waterfall era. And so autonomy is a key concept, again, in terms of better software outcomes. And, yeah, having teams and individuals doing their best work. Right. And because what you all say in the book is, like, you don't want people blocked from resources, right? Alex Williams You want them to be able to, like, do their work independently, to do that work autonomously. And if they're needing permissions or if they're needing, you know, if they have to wait for somebody, that just, like, creates that jerk again. ([Time 0:13:52](https://share.snipd.com/snip/b63f47da-a7c0-42c8-b7b2-e376de50602d)) - **Balance Autonomy With Business Alignment** - Alignment balances autonomy with business goals so release decisions reflect product priorities. - Radical delegation can let product managers control broad rollouts while developers push changes continuously. Transcript: Alex Williams Yeah. Alignment means focusing human and organizational resources responsible for developing software to work in the same direction. James Governor Yeah. Well, alignment, and, you know, we've had areas. I mean, we had this thing called ITIL. Some people have some battle scars. The younger listeners certainly won't know what the hell I'm talking about. But the idea was that you could have alignment in terms of discussing or in terms of delivering. Yeah, ITIL was basically a common language that would enable you to have alignment between the business and IT. Now, that may not have worked in a sense, but that's exactly what we're trying to do. We're trying to have better outcomes. You want alignment between the user, preventing the jerk that you mentioned, and the software they're using. But certainly, how do you get better alignment between the business and the people building the software? So for example, one of the concepts we have of sort of radical delegation is, okay, well, you do want autonomy, but by the same token, you may want the product manager rather than the software Developer to be the person that actually decides when the release happens. So you could be shipping software, you could be building software, but whether you're actually releasing that to the entire user cohort, that's a decision for the business rather Than the individual developer. So we just wanted to dig into that, the different kinds of alignment. And I was lucky enough. And it's just between autonomy, it's just between abundance and autonomy. It really does, yeah. And I think alignment, it was a really interesting conversation earlier with Amazon. So Amazon is on a journey. They're one of the case studies in the book. So Singh and Claire Liguri, who keep popping up, they're like the zealot of cool things at Amazon, right? So, you know, built out their container story. Now Deepak is working on their developer tools. There's a new tool, Kiro, that they've got. Alex Williams He's one of the people who helped develop that. James Governor It's his team. It's his team. And he is a visionary. I mean, he's quite hands-on. Yeah. But in talking to them in terms of a case study, that was really interesting because Amazon, I mentioned the two-piece team, historically, the service, because for them, it is all about, Yes, they talk about customer obsession, but it's all about the service teams. Who's delivering the service? They're making the engineering decisions. They're choosing the tooling that will deliver a product, deliver the service to the end user. So there was a sort of, it becomes a challenge of alignment because if different teams are doing different things and making different choices, that can make things more difficult From a software delivery perspective. Different companies have this to varying degrees. Obviously, you take a company like Google, very, very standardized. Very standardized in terms of the entire tooling that anyone that is building in the company uses. So for Amazon, they sort of had a bit of an alignment problem because they had given so much autonomy. So you're trying to balance these things against each other. ([Time 0:15:51](https://share.snipd.com/snip/0a1878a6-e4d8-4617-9f5b-80d0cecdac7c)) - **Amazon Balances Autonomy And Standardization** - James Governor cites Amazon as a case study wrestling with autonomy versus standardization across teams. - Amazon is moving toward more standardized tooling and a "cost to serve" measure to improve alignment. Transcript: James Governor In order to deliver these services to end users? These are alignment questions. And organizations need to be thinking about how do they balance that. So yeah, I mean, they've got sort of an interesting new framework they're developing internally at Amazon about the cost to serve the customer, right? And this is something that came from their logistics operations, but they're now using it internally as a measure of like, yeah, how do we understand everything that is involved in Everything comprised in delivering service to energy it was very interesting to me that they are absolutely living this progressive delivery journey in their own right and solving In fact some issues the book was out of date they already you know i talked to amazon today and it's like oh thanks you ([Time 0:19:56](https://share.snipd.com/snip/5b028cde-67a1-4793-8cf7-7f57391f13e7)) - **Progressive Delivery Is Essential For AI** - AI amplifies uncertainty: models hallucinate and behave non-deterministically in production. - Progressive delivery practices—evals, versioning, measurement, rollback—are essential for safe AI deployment. Transcript: James Governor Hundred percent. And, oh, damn it, I said it again. So, cut. So, the thing about, one thing I find is super interesting, and again, this AI explosion, you know, we were already sort of, we'd already worked on a lot of concepts. The book took a while, you know, and it was sort of interesting when we got the call from ITREV and they were like, okay, let's go ahead with this. We were kind of thinking, wait, do we need to have a fifth aid? We need to like really focus on AI because it's having such a profound effect on software delivery. But I think actually what's true is the concepts, as you say, are longer lived. And in fact, if you look at what people are building, when they talk about evals, from a, you know, you're rolling something out, you're not exactly sure how it's going to behave, how The system will behave, you might change the model, what impacts is that going to have? We need to measure that. We need to see its actual behavior. Then think about doing a rollback. All of those things, that's progressive delivery. Interesting, Thomas Domke, when he was running GitHub, I talked to him, and he did. He said, look, you can't do AI-based development without progressive delivery. Right. Alex Williams You talked about that today. The last A in the four A's is automation. And automation is the identification and implementation of programmatic processes for repetitive tasks, which follows alignment. Absolutely. 100%. I said it. There you go. James Governor Cut. This is what we do in IT. Like, this is what we do in software. You talk about repeatable processes. If we can take that and automate it, that makes us more effective. And I think when you ask the question about what's different, or, you know, why did you come up with this framework? It was very clear. Look, automation is not new. That is literally the oldest thing in IT. Or civilization. Or civilization. However, the tooling that we've currently got, we think about the cloud abundance, we think about this open source availability, the tooling for automation is so much more sophisticated Than it ever was. The config as code revolution, I think, really took us forward. And certainly some of those concepts were absolutely essential in being able to do the kinds of things that we're talking about with progressive delivery, where you were able to route Traffic, you were able to make changes, you were able to do rollbacks, you're able to understand user behavior, bring that into the process. All of that requires a lot of automation. And so, yeah, that's not new. And certainly it's not going to be, that's not new. And certainly it's going to be massively applicable in this AI revolution. But yeah, it's absolutely core to better software delivery. You want to be able to automate those. I mean, yeah, what are we doing here and becoming more effective? It's always about automation. Alex Williams Well, James, I think that's a good place to conclude unless there's anything else you'd like to add about the book, about progressive delivery. We've gone through the four A's. We were going to say the fifth A, which is AI. I think we've covered that pretty well. But what I like about what you said about AI, if I may recap, that a lot of the practices we see in AI have their roots in software development life cycles that we see through the work of Edith Harbaugh, for instance, rollbacks and such, right? And so that seems to me like where we're going with progressive delivery is those concepts. You're going to still need versioning of some kind. James Governor Absolutely. And look, I mean, if we just think about behaviors and when you're trying to teach and learn, which is kind of what we're doing with these large language models. And as we've seen again and again and again, sometimes they have behaviors that are really unhelpful socially. Yeah. You know, whether it's the language they use, you know, you've got a platform that, you know, is perhaps insulting. Now there are those that would celebrate that. But, you know, I think most organizations that are building software and caring about users actually do, you know, if they've got a model that's hallucinating more, they want to roll It back. They want to think about what they're doing. That being intentional absolutely requires, as you say, that these disciplines that we've built over decades, over the history of IT, in effect, are applicable. ([Time 0:22:10](https://share.snipd.com/snip/bc79a3d1-0f79-415b-82f7-173b6fac4dda)) - **Automate Rollouts And Rollbacks** - Automate repeatable delivery tasks like routing, rollbacks, and measurement to scale progressive delivery. - Invest in config-as-code and tooling to detect user behavior and perform controlled rollouts automatically. Transcript: Alex Williams You talked about that today. The last A in the four A's is automation. And automation is the identification and implementation of programmatic processes for repetitive tasks, which follows alignment. Absolutely. 100%. I said it. There you go. James Governor Cut. This is what we do in IT. Like, this is what we do in software. You talk about repeatable processes. If we can take that and automate it, that makes us more effective. And I think when you ask the question about what's different, or, you know, why did you come up with this framework? It was very clear. Look, automation is not new. That is literally the oldest thing in IT. Or civilization. Or civilization. However, the tooling that we've currently got, we think about the cloud abundance, we think about this open source availability, the tooling for automation is so much more sophisticated Than it ever was. The config as code revolution, I think, really took us forward. And certainly some of those concepts were absolutely essential in being able to do the kinds of things that we're talking about with progressive delivery, where you were able to route Traffic, you were able to make changes, you were able to do rollbacks, you're able to understand user behavior, bring that into the process. All of that requires a lot of automation. And so, yeah, that's not new. ([Time 0:23:30](https://share.snipd.com/snip/40b5dc49-b6a9-4906-bd76-3a8cb9d79f03))