Why Good Teams Rely on Bad Workarounds - CourtneyR.dev's 2nd Brain

Why Good Teams Rely on Bad Workarounds ![rw-book-cover](https://img.youtube.com/vi/BII17aShQH0/maxresdefault.jpg) 🔗 **Source:** [Why Good Teams Rely on Bad Workarounds](https://www.youtube.com/watch?v=BII17aShQH0) > [!note]+ Document Notes > tags: #devops #feedback-loops #platform-engineering #developer-experience #continuous-integration #team-dynamics > author: Lead Dev > source: YouTube > reference: https://www.youtube.com/watch?v=BII17aShQH0 > date created: 2024-12-19 10:30 > > # 🏷️ Title: Why good teams rely on bad workarounds >  ## 🧰 Action Inbox - [H1] Audit build and deployment times across teams — identify normalization of 4+ minute delays #developer-experience - [H2] Implement team hack weeks to address accumulated technical friction — prioritize flaky tests and manual workarounds #platform-engineering - [H3] Create developer feedback surveys treating platform as product — measure actionable pain points #devops ## 🧭 Table of Contents - [[#📝 Summary]] - [[#📖 Main Content]] - [[#🗒️ Notes]] - [[#📚 References]] - [[#🗣️ Paraphrase]] - [[#📰 Newsletter]] - [[#✅ Action Items]] - [[#🧪 Hypotheses]] - [[#📊 Methodology]] - [[#❓ Questions]] - [[#🔮 Future Research]] - [[#💡 Implications]] - [[#🧠 Learning Styles Assessment]] - [[#📊 Accessibility Assessment]] - [[#🌍 DEIB Assessment]] - [[#✅ Logical Validity]] - [[#🛡️ DARVO Analysis]] - [[#🎯 Conclusion]] ### 🔗 References - [Lead Dev Online Panel](https://www.youtube.com/watch?v=BII17aShQH0) - [DORA Research](https://dora.dev) ## 📝 Summary - Engineering panel discussing how delivery dysfunction stems from broken feedback loops rather than process/people issues - Teams normalize slow builds, flaky tests, and manual workarounds due to competing priorities and lack of ownership - Key insight: feedback loops should help discover issues earlier, but teams often mistake intermediate metrics for customer satisfaction - Panelists emphasize treating internal platforms as products and connecting developers directly to user feedback - Bottom line: systemic improvements in feedback speed require cultural changes, not just tooling fixes ## 📖 Main Content This Lead Dev panel features engineering leaders exploring why teams tolerate broken delivery systems. The core argument is that delivery problems are often systemic rather than procedural - teams adapt rationally to slow builds and unreliable tests, but these adaptations create long-term structural costs. The panelists define feedback loops as mechanisms to discover earlier what would eventually be discovered through customer interaction. Steve Fenton uses the sandwich metaphor: until a customer takes a bite, you don't know if you've made what they want. However, waiting for that final feedback is too slow, so teams create intermediate loops through testing and staging. Common normalized friction includes flaky tests (often just commented out), slow repository clones (4+ minutes accepted as normal), and environment drift between staging and production. Teams tolerate these because the system still functions - it's just inefficient. The discussion reveals that metrics can distort feedback when pursued in isolation. Teams might remove tests to meet deployment speed targets or optimize for vanity metrics while losing sight of user outcomes. The panelists argue for connecting developers directly to users and treating internal platforms as products with real customer needs. ## 🗒️ Notes #### ⏱️ [00:00](https://youtu.be/BII17aShQH0?t=0) — [H1] Introduction to feedback loop dysfunction - Key idea: Delivery problems are systemic, not just about process and people - Evidence: "Teams tend to adapt... those adaptations are often rational but cost money long-term" #### ⏱️ [05:30](https://youtu.be/BII17aShQH0?t=330) — [H2] The sandwich metaphor for feedback loops - Key idea: Intermediate feedback loops should predict eventual customer satisfaction - Evidence: "Until I give you that sandwich and you take a bite, we don't know if I've made something you want to eat" #### ⏱️ [08:45](https://youtu.be/BII17aShQH0?t=525) — [H3] Developer experience matters for feedback effectiveness - Key idea: Slow feedback disrupts developer flow and mental models - Evidence: "Repository cloning taking 4+ minutes was normalized, reduced to 26 seconds with shallow clone" #### ⏱️ [12:20](https://youtu.be/BII17aShQH0?t=740) — [H2] Common pipeline friction: flaky tests and environment drift - Key idea: Teams normalize small annoyances until they compound into major problems - Evidence: "Just run it again, it'll succeed the second time... nobody's had time to dig into it" #### ⏱️ [16:10](https://youtu.be/BII17aShQH0?t=970) — [H3] Environment parity challenges - Key idea: Different deployment processes between environments cause production failures - Evidence: "Developers automate to dev/test, but traditional ops does production deployment differently" #### ⏱️ [19:30](https://youtu.be/BII17aShQH0?t=1170) — [H1] Metrics can distort rather than improve feedback - Key idea: Optimizing metrics in isolation can harm user experience - Evidence: "Deploying more often is good, but daily app update notifications become irritating to users" #### ⏱️ [23:45](https://youtu.be/BII17aShQH0?t=1425) — [H2] Why teams tolerate broken feedback loops - Key idea: Normalization of deviance combined with roadmap pressure - Evidence: "Internal work doesn't give visible progress on roadmap, gets deprioritized" #### ⏱️ [27:15](https://youtu.be/BII17aShQH0?t=1635) — [H3] Lack of clear ownership across boundaries - Key idea: Pipeline problems span code, platform, and infrastructure with no single owner - Evidence: "Bot PRs fail with linter issues, requiring manual intervention each time" #### ⏱️ [31:20](https://youtu.be/BII17aShQH0?t=1880) — [H2] Justifying pipeline improvements with business metrics - Key idea: Convert engineering pain into dollar-valued impact for leadership buy-in - Evidence: "Hours spent on slow builds and flaky tests translates to money lost" #### ⏱️ [34:50](https://youtu.be/BII17aShQH0?t=2090) — [H1] Professional responsibility for technical quality - Key idea: Engineers should make quality decisions without asking permission - Evidence: "Plumber doesn't ask permission to use copper pipes - they're the professional" #### ⏱️ [37:40](https://youtu.be/BII17aShQH0?t=2260) — [H1] Direct user feedback as untapped resource - Key idea: Connecting developers to users improves product team performance - Evidence: "User centricity brings performance to highest levels even with slower deployment frequency" #### ⏱️ [40:25](https://youtu.be/BII17aShQH0?t=2425) — [H2] Platform teams must treat platform as product - Key idea: Internal customers deserve same research and feedback attention as external users - Evidence: "Survey internal customers, don't assume you know what they need" #### ⏱️ [42:15](https://youtu.be/BII17aShQH0?t=2535) — [H3] Sustainable feedback loops require human consideration - Key idea: Metrics optimization can create unsustainable work environments - Evidence: "Customer service optimized for ticket closure speed creates copy-paste responses" ## 📚 References - [DORA Research Community](https://dora.dev) - [Lead Dev Platform](https://leaddev.com) - [Octopus Deploy](https://octopus.com) ## 🗣️ Paraphrase - Engineering delivery problems often stem from broken feedback systems rather than inadequate processes or insufficient discipline - Teams rationally adapt to slow builds and unreliable tests, but these workarounds create hidden long-term costs - Effective feedback loops help teams discover issues earlier in development rather than waiting for customer complaints ## 📰 Newsletter **Platform Engineering Insights: Why Good Teams Rely on Bad Workarounds** This week's Lead Dev panel revealed a counterintuitive truth: high-performing engineering teams often develop sophisticated workarounds for broken systems rather than fixing root causes. The discussion highlighted how 4-minute repository clones become "normal" and flaky tests get commented out instead of debugged. Key takeaway for WordPress teams: treat your development platform like a product. Survey developers about pain points, measure build times as business metrics, and create dedicated time for infrastructure improvements. The panelists emphasized connecting developers directly to user feedback - a practice that improves team performance even when deployment metrics lag behind competitors. Resources: [Learn WordPress](https://learn.wordpress.org/), [Developer Docs](https://developer.wordpress.org/), [WP-CLI](https://wp-cli.org/) ## ✅ Action Items #### DevRel - Developer experience: audit CI/CD pipeline performance across community projects, create benchmarks for acceptable build times - Education: workshop content on feedback loop optimization, including metrics selection and normalization awareness - Community management: survey developers about normalized friction points, prioritize based on impact and effort #### OSPO - Developer Support: tooling consultation for faster feedback loops, CI/CD best practices documentation - Community Engagement: fund improvements to critical project infrastructure, sponsor hack weeks for maintenance - Strategic Alignment: KPIs that balance speed metrics with developer satisfaction and system reliability #### Community - Maintainers: governance templates for infrastructure investment decisions, roadmap balance between features and developer experience - Contributors: onboarding improvements focused on reducing time-to-first-contribution through better tooling #### WordPress - Core: build time optimization initiatives, flaky test identification and remediation programs - Core Performance: profiling CI/CD bottlenecks, metrics for developer productivity impact - Community: developer experience surveys, infrastructure improvement showcases at WordCamps - CLI: command performance optimization, developer workflow improvement tools - Meta: .org platform reliability metrics, API response time optimization for developer tools ## 🧪 Hypotheses - Teams that measure and celebrate infrastructure improvements show higher developer satisfaction and retention rates - Organizations with direct developer-to-user feedback channels deliver features with higher adoption rates despite potentially slower deployment frequency - Platform teams that treat internal developers as customers see reduced support requests and higher platform adoption - Regular "infrastructure hack weeks" reduce accumulated technical debt more effectively than continuous background maintenance ### 📊 Methodology - Survey methodology comparing developer satisfaction before/after infrastructure investment programs - A/B testing framework for measuring impact of direct user feedback on feature success rates - Longitudinal study design tracking technical debt accumulation patterns across teams with different maintenance approaches ### ❓ Questions - How do we measure the business impact of normalized developer friction without creating new metric distortions? - What organizational structures best support shared ownership of cross-boundary infrastructure problems? - How can leadership teams balance roadmap pressure with necessary infrastructure investment in measurable ways? ### 🔮 Future Research - Quantitative analysis of correlation between build performance and developer retention across organizations - Investigation of cultural factors that enable teams to prioritize infrastructure work without external pressure - Cross-industry comparison of internal platform success metrics and developer satisfaction outcomes ### 💡 Implications - Practice: Regular infrastructure investment prevents expensive workaround accumulation and developer burnout - Policy: Platform teams need clear customer research processes and success metrics beyond uptime - Education: Engineering leaders require training on recognizing and addressing normalization of deviance - Funding: Infrastructure improvements should be justified using developer productivity and business impact metrics ### 🧠 Learning Styles Assessment Reading level estimate: 12th grade #### 👀 Visual learners - Present: Discussion format with multiple expert perspectives - Improve: Workflow diagrams showing feedback loop breakdowns, before/after pipeline visualizations #### 🔊 Aural learners - Present: Full 45-minute panel discussion with multiple voices and perspectives - Improve: Key takeaway summary audio, actionable tips in podcast format #### 🗣️ Verbal learners - Present: Rich analogies (sandwich metaphor), clear technical term definitions, real-world examples - Improve: Glossary of platform engineering terms, case study narratives #### 🧩 Kinesthetic learners - Present: Concrete examples of pipeline improvements with specific timing improvements - Improve: Step-by-step pipeline optimization checklists, hands-on feedback loop assessment tools #### 🧠 Logical learners - Present: Cause-and-effect relationships between metrics and outcomes, systematic problem analysis - Improve: Decision trees for infrastructure investment prioritization, ROI calculation frameworks #### 🧑‍🤝‍🧑 Social learners - Present: Panel format with expert interaction, follow-up Slack discussion encouraged - Improve: Team retrospective guides for feedback loop assessment, peer review processes for infrastructure improvements #### 🧘‍♀️ Solitary learners - Present: Individual reflection prompts about team practices and normalization patterns - Improve: Self-assessment questionnaires for pipeline health, personal productivity tracking worksheets #### 🌿 Naturalistic learners - Present: Systems thinking about engineering ecosystems and organizational health - Improve: Analogies to natural feedback systems, sustainability metaphors for technical debt ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Video content with spoken discussion, clear speaker identification - Improve: Full transcript with speaker labels, heading structure for navigation #### Keyboard navigation - Present: Standard YouTube controls accessible via keyboard - Improve: Timestamp navigation shortcuts, chapter markers for topic jumping #### Contrast and readability - Present: Standard video player interface with adequate contrast - Improve: High contrast mode options, adjustable text size for any overlays #### Media and captions - Present: Auto-generated captions available on YouTube - Improve: Professional captions with technical term accuracy, downloadable transcript #### Standards reference - Meets WCAG 2.2 AA for video content with available captions and keyboard controls ### 🌍 DEIB Assessment #### Cultural breadth - Present: International perspectives with panelists from US, Finland, and Brazil - Improve: Examples from different organizational structures and company sizes globally #### Inclusive language - Present: Gender-neutral technical discussions, professional terminology explained - Improve: Avoid idioms that don't translate across cultures, define regional-specific practices #### Neurodiversity support - Present: Structured discussion format with clear topic transitions - Improve: Topic timestamps for focused listening, visual agenda or outline #### Representation - Present: Diverse panel including women and international speakers in leadership roles - Improve: Broader range of company sizes and industry verticals represented #### Psychological safety - Present: Respectful discussion of common engineering challenges without blame - Improve: Acknowledgment that infrastructure struggles are normal and addressable ### ✅ Logical Validity #### Strong reasoning - Deductive strength: Logical progression from feedback loop theory to specific pipeline problems - Inductive strength: Multiple examples supporting general principles about normalization of deviance - Causal reasoning: Clear connections between slow feedback and decreased developer productivity - Analogical strength: Effective sandwich metaphor illustrating feedback loop concepts #### Fallacies - Appeal to authority: Panelists' credentials support but don't prove their claims - mitigated by concrete examples - Hasty generalization: Some broad claims about "teams" based on limited examples - addressed through multiple perspectives #### Tone of voice - Overall tone: Collaborative and solution-focused, with appropriate technical authority and practical empathy for common engineering challenges ### 🛡️ DARVO Analysis - Deny: No instances of denial; panelists acknowledge widespread infrastructure problems - Attack: No attacks on accusers; discussion focuses on systemic rather than personal issues - Reverse victim/offender: No victim/offender dynamics present in technical discussion - Emotional cues and rhetoric: Professional, empathetic tone recognizing shared struggles without blame ## 🎯 Conclusion The panel reveals that engineering delivery problems often stem from rational adaptations to broken feedback systems rather than inadequate processes. Teams normalize slow builds, flaky tests, and manual workarounds because fixing them requires cross-boundary ownership and competes with feature delivery pressure. The key insight is treating infrastructure improvements as professional responsibilities and internal platforms as products requiring user research. Success requires cultural changes that celebrate system improvements alongside feature delivery, measure developer productivity as business impact, and create dedicated time for addressing accumulated technical friction. ## 🧱 Refactor Appendix ### 📚 Literature Split Pad ### [H1] Feedback loops predict customer satisfaction through early discovery > [!quote] Why good teams rely on bad workarounds by Lead Dev via YouTube. Published 2024. Accessed 2024-12-19. > "Until I give you that sandwich and you take a bite, we don't know if I've made something that you want to eat." Core principle that intermediate feedback should reliably predict final user satisfaction rather than becoming goals themselves. [ --- ## Full Transcript [00:12] Hi everyone, welcome to this episode of [00:16] of lead panel. Um is this is like great [00:20] today because we have like a great [00:22] session on how feedback look uh fail and [00:25] how to fix them. Um this is a lead dev [00:27] online panel discussion created with [00:29] partnership with octopus deploy. Um [00:32] discussion will roughly take 45 minutes [00:34] I think uh give or take after we're [00:37] going to move into slack. So if you're [00:38] interested you have some follow-up [00:40] question please join uh me and the [00:42] panelist in Slack uh is there is like a [00:46] panel Q&A channel so you can easily get [00:48] there and uh if we have time hopefully [00:51] we'll have we can also some answer some [00:54] question online. Um, so there is a Q&A [00:57] functionality in Zoom. Feel free to use [00:59] it and post your message. If it's [01:01] relevant to the current topic, I I I'll [01:03] pick it up in a minute. If not, we're [01:04] going to try to do at the end. If not, [01:06] we're gonna respond in the channel. So, [01:08] it's going to be you you will have your [01:10] answer. Don't worry. And let me start [01:13] with some intro. So, my name is Simony [01:16] uh and I'm a head of engineering at a [01:18] company called Kraken. And today I'll [01:20] join by like an incredible list of [01:22] panelists. I had a very exciting [01:24] discussion with them the other day. Uh [01:27] Steve, Fran, Francine. So um I'll start [01:31] ask shut up and I'll let them introduce [01:33] themselves. So Steve, do you want you [01:36] are on my right, so do you want [01:37] introduce yourself first? [01:39] >> Hi everyone. Uh yeah, my name is Steve [01:40] Fenton and uh I am a uh Devril and [01:44] researcher at Octopus Deploy and a [01:47] member of the Dora community um which is [01:49] Google's research arm and uh yeah, I'm [01:52] super excited to be talking about [01:53] pipelines today. [01:56] >> Awesome. Ren, [01:59] >> hey everyone. I'm Ren Daniels. I'm the [02:01] tech lead of the cloud platform team at [02:04] Mapbox and I am based out of Helsinki, [02:06] Finland. [02:08] >> Nice. and Francine. [02:11] >> Hi everyone, I'm Francine Pilot. Quickly [02:15] describing myself here. I'm a white [02:17] woman, long dark brown hair wearing a [02:19] black t-shirt and I'm a lead software [02:22] engineer at New Bank working on the [02:24] CI/CD platform that supports our [02:27] engineer teams to building reliable [02:29] pipelines and deploying at scale. I'm [02:31] really happy to join the conversation [02:33] today [02:36] >> and we are too we are to 100%. So uh let [02:41] let's give the audience a bit of a [02:42] context. So this is how we decided to [02:44] frame the conversation and then I'm [02:45] going to have space for the panelist to [02:47] chime in. Uh there is a lot of content [02:50] to cover I think. So when we talk about [02:53] delivery program uh in my mind we often [02:55] default to process and people. The kind [02:57] of thing the kind of comment you get in [02:58] the room are we need better planning, we [03:00] need more discipline. [03:02] You may hear the old school we need more [03:04] engineers or the engineer are not [03:06] working hard enough. Um but in reality [03:10] uh also a lot of delivery dysfunction [03:11] are systemic and not necessarily about [03:14] what I described and are inside what we [03:17] call our feeder loop. Our ability to [03:19] learn how far we are from where we want [03:21] to go and um example are you know if [03:24] your build takes 40 minutes uh if when [03:28] the tests are failing when deployment [03:30] painful or even when some of the metrics [03:33] are misleading if we talk about customer [03:34] feedback uh then team tend to adapt. [03:37] they kind of like they adjust to the [03:38] fact that there are things that are a [03:40] bit Sorry for for the sword. Uh [03:44] and those adaption are often rational. [03:46] It kind of makes sense. Uh but I think [03:49] in the long term they are going to cost [03:51] the company quite a lot of money. So [03:53] today I would us to explore where [03:56] feedback loops normally break down. um [04:00] what are the quick fixes that create [04:03] some structural drug let's call [04:04] structure I like structural drug and [04:07] what modern engineering leadership [04:09] leaders can do systematically not just [04:12] procedurally to restore let's call it [04:14] flow um so this is roughly what I want [04:18] to talk about and I think I cannot not [04:20] start from think about feedback loop so [04:23] Steve we we talked about this so I [04:25] wanted your take on it what when you [04:27] think about feedback loop in delivery. [04:30] What are the most overlooked ones? Where [04:32] do teams misunderstand [04:34] um what where the what a figure loop is? [04:38] Yeah. Uh this is an interesting [04:40] question. I think um if we were going to [04:43] be really hardcore about it, we'd [04:45] probably say there's one real feedback [04:46] loop, right? If you ask me to make you a [04:48] sandwich, until I give you that sandwich [04:51] and you take a bite, we don't know if [04:53] I've actually made something that's that [04:54] you want to eat, right? Um, but we also [04:58] know that that sometimes takes too long. [05:01] Like if I make a small code change and [05:02] then I have to wait for a customer to [05:04] tell me, then I'm going to be kind of [05:07] like behind the curve. So the idea is [05:10] that feedback loops really ought to be [05:13] um ways of us discovering earlier [05:15] something that we would eventually [05:17] discover. So the reason we do testing is [05:20] because you know I remember when I first [05:23] started in my career we would put a [05:24] version out and people would say I can't [05:26] even log into this version right and so [05:28] we're kind of getting really lowquality [05:30] feedback from that big real feedback [05:32] loop um so we've tried to create tighter [05:35] and tighter loops inside [05:37] um to try and get earlier sight of [05:40] things and to make sure that what [05:42] eventually comes out and that customers [05:43] bite into is tastes good. Um, but I [05:46] think probably one of the biggest [05:47] mistakes is mistaking those intermediate [05:50] pieces of feedback for someone being [05:52] delighted with the sandwich. [05:56] >> This is brilliant. I wish I had the the [05:58] love of BJI to to spark in the [06:00] conversation. I love the metaphor. Thank [06:02] you so much. [06:02] >> Sorry if I've made everyone hungry. [06:08] want someone else want to jump in to to [06:10] have their own take on people loop and [06:12] if you see it differently or similarly [06:16] >> yes I I would like to add something here [06:19] is about the developer experiency also I [06:24] think that [06:26] what I seen in in practice like when we [06:30] tend to focus on build success or [06:32] deployment status [06:35] uh we cannot ignore how take it uh takes [06:41] to get feedback and how actionable it [06:44] is. And ju just to picture what I'm [06:48] trying to say is once I worked on a [06:51] pipeline where just cloning the [06:53] repository was taking over four minutes [06:56] and it has been normalized. Nobody was [06:59] question it. just be people just wait [07:02] for it to finish and then what Stevie [07:06] said about getting to the the customer [07:09] and waiting for so long to to taste it. [07:13] But we can just tight this little uh [07:19] space in the pipeline little step by [07:23] maybe just changing why the cloning [07:26] strategy is used like doing a shallow [07:29] clone and in this particular case we [07:32] drop to 26 seconds. So it goes to the [07:36] four to 26 seconds. uh so I I think that [07:39] we can not just improve the performance [07:43] and maybe help how uh people during the [07:48] work day are are changing and are [07:51] working and the green check is not [07:56] exactly the health feedback loop if it [07:59] is just too is low we need to improve it [08:02] it and pay attention on those things [08:05] >> yeah this is super crucial because when [08:07] I'm working on the actual code part of [08:10] the problem. I don't want to have to [08:13] pause and wait to get that in a loop [08:15] feedback. I kind of want that to be [08:17] instant cuz otherwise my brain starts to [08:20] let go of all of this model that I've [08:22] built and what I'm doing and the stages. [08:25] So yeah, I I don't want to wait like 40 [08:28] seconds to to do a a kind of clone. I [08:31] don't want to wait like 20 minutes for a [08:33] build to complete. It's just too long [08:34] for me to keep in that flow. [08:38] Interesting. Interesting. Basically what [08:40] what you're saying is that obviously [08:41] Steve started by saying well uh this is [08:44] the feedback loop is the one with the [08:45] customer but then Francie point is like [08:48] well the feedback loop of the engineers [08:49] also matter. It's kind of like they get [08:51] so um it's super interesting super [08:55] interesting. Um and what you mentioned a [08:58] bit Francine but I wonder if someone [08:59] else has I don't know for example Ren I [09:01] think you I would love your take on [09:04] this. What are the most common form of [09:06] pipeline friction that the team [09:07] normalize? So friends mentioned one for [09:10] sure but uh are and uh are actually to [09:14] sign or signal of a deeper issue or of [09:18] deeper issues in your mind. [09:21] >> I think probably the the biggest one [09:24] that I've seen or the most common one [09:25] that I've seen is the tests that are [09:28] flaky where oh yeah it failed that time [09:31] just run it again it'll succeed the [09:33] second time. We don't know why nobody's [09:35] ever had the time to really dig into it [09:38] because at least on practically every [09:41] engineering team that I've ever worked [09:42] on, there's never enough time and [09:44] there's always something that's higher [09:45] priority. So when you have these little [09:47] annoyances like the little paper cuts [09:49] like, you know, the the build that's [09:51] just a few seconds too long or the tests [09:53] that are flaky every so often, it's not [09:56] enough of a priority to like, all right, [09:58] stop the line. Let's dig in and really [10:00] figure out what's going on. but just [10:02] these little annoyances that you never [10:03] really notice until somebody new joins [10:05] your team and then you have to explain [10:07] them to somebody and you're like that's [10:09] kind of embarrassing. Why didn't we fix [10:11] that? Um, so I think it can be really [10:13] important, especially when we're talking [10:15] about these internal feedback loops, to [10:17] just think about ways of capturing that [10:21] information. And one thing that I like [10:23] to do is like do a team hack week every [10:25] so often where you you know prioritize [10:27] fixing those things that nobody's had [10:29] time time or priority to fix before. [10:32] >> Yeah. And the number one fix for flaky [10:34] tests is that people just comment them [10:36] out like and I don't have any actual [10:38] research or evidence for that but I've [10:39] opened up a lot of code bases where [10:41] there's lots of tests that are just [10:42] commented out and it's always because [10:44] they kept on kind of flip-flopping [10:46] between green and red and it was just [10:48] too much effort to try and fix it. Um, [10:50] so yeah, that that resonates. I think [10:53] that's a that's a major problem. [10:57] >> So, so Danny, it sounds like that what [10:59] you're saying is um slow and flaky are [11:03] the two main common [11:06] >> or just less useful. Like one thing I've [11:08] also seen is a ongoing drift between, [11:11] you know, a staging and a production [11:12] environment where ideally those are are [11:15] pretty similar. So if you deploy [11:16] something to staging, you know, it's [11:18] probably going to work the same way in [11:20] production, but then drift happens and [11:23] you end up with the same sort of thing [11:24] where you lose confidence in the system. [11:27] It's not doing what it's supposed to do, [11:29] which can also, you know, the the loop [11:32] is giving you incorrect feedback. So [11:34] that's not it's not doing its job there. [11:36] Yeah, I've I've seen a a problem that [11:38] happens quite a lot where um the [11:41] developers like will automate their [11:43] deployments to dev and test, but then [11:46] there's a more traditional ops team that [11:48] wants to do the production deployment. [11:50] And so you then end up in a situation [11:53] where you're like you get this [11:54] confidence dev worked fine, test worked [11:56] fine, and then the production deployment [11:59] doesn't work because it's not done in [12:01] the same way that the dev and test [12:02] deployments were done. Um, so that's a [12:05] really common problem and I've I've hit [12:07] that so many times in my career. Um, [12:09] where we've solved the problem but can't [12:13] complete that all the way into [12:14] production. Um, you know, only takes a [12:17] tick box to be missed. I worked on one [12:19] application where they did the whole [12:20] checklist apart from double clicking on [12:22] the installer. So there was actually no [12:24] software installed as part of their [12:26] whole deployment process. So yeah, you [12:30] kind of need to make sure your [12:31] environments are reasonably similar. Um, [12:34] if you use load balancing in production [12:36] and you're not losing using load [12:38] balancing in test, you're going to find [12:40] little things like, oh, we were using [12:41] session state and as I move to a [12:43] different server, the session state is [12:44] not there. Oops, we should have used our [12:46] redis cache or whatever. So yeah, you [12:48] kind of want to have that similarity all [12:50] the way through. Um, and like if you're [12:53] changing your process for deployments, [12:55] you want to make sure that it follows [12:57] the same path to production. We're [12:59] really good now at making sure my code [13:00] change goes to dev and then to test and [13:03] then to production, but sometimes we [13:05] change our deployment process. We're [13:06] like, oh, when we deploy to test, we [13:08] miss something. We'll we'll update the [13:10] process for it or we'll change how it [13:11] works. And then the next deployment you [13:13] do is production on this process you've [13:16] never even used before. It's like the [13:18] first time you've used it. So you kind [13:20] of want to flow that through the same [13:22] path like oh the process didn't work [13:24] we'll update the process then we'll [13:26] deploy to dev again and then to test and [13:28] now we've tested our process twice as [13:30] well as the application. [13:33] >> Yeah makes sense. I mean I have a bit of [13:35] a spicy take here so I'll put I'll put [13:38] it down right. Um, I seen so many teams [13:43] spending I was going to say wasting but [13:45] maybe it's too strong so much time [13:47] trying to make a staging equal to [13:51] production instead of like a building [13:54] capability to allow them to actually [13:55] test where it really matter that is the [13:57] production environment and so I um I [14:01] wonder if the people have this anyway [14:03] this is what we do at Kraken we don't [14:04] have a staging environment we try to [14:06] define a way that is safe in production [14:08] so that we can do some you know some [14:10] testing there and then we're roll out to [14:13] uh other clients. Um I'm wondering how [14:16] much uh is this something you explored [14:18] as well or [14:21] >> Yeah, I mean my my view on that is that [14:24] um it should be the same shape as [14:26] production, your staging environment, [14:28] but it can be miniature, right? So you [14:30] probably don't need to have your full [14:33] spec like of everything running, but if [14:37] you're doing things like load balancing [14:39] or if you have caches and things like [14:41] you want all of that stuff to be there [14:43] in miniature and to be cheap and ideally [14:46] easy to set up with automation. Like I [14:49] think we're past the point where we want [14:51] to be clickopsing our environments cuz [14:54] they just like 5 minutes later you have [14:57] a completely different environment to [14:58] what you expected cuz things change. so [15:00] rapidly. So we want to automate those [15:02] environments and we want to have them in [15:04] miniature but we also know that testing [15:07] itself is miniature. We might run a [15:09] thousand tests but then when we put our [15:12] software live there's like it's not [15:15] going to be a thousand people using it. [15:16] It could be millions of people using it [15:18] and they do all kinds of things we can't [15:20] anticipate. And so we need to that's [15:23] where that observability of what's [15:24] happening in production becomes so [15:25] useful because users create ideas of how [15:29] to use your software that you never [15:30] anticipated and it can make it go um [15:33] wrong you know [15:37] >> thank you. Um [15:40] thank you. Um I'm wondering no one has [15:44] uh you know said the uh the M word yet [15:48] that is metrics and I'm surprised [15:51] because it come out after 30 seconds [15:53] normally. So uh so instead of you asking [15:56] the usual question I want to ask like a [15:58] contrarian question is where actually [16:00] where do measurement you know of system [16:03] actually distort feedback instead of [16:04] improving it. Is there is there like [16:07] some some situations some scenario where [16:10] actually sometimes I do a disservice [16:12] more than helping [16:17] >> Steve I know you see me let's go with [16:20] you [16:20] >> I I always when metrics comes up I [16:22] always try to pause for a minute because [16:23] I know I get over excited about the [16:25] subject of metrics there's loads of [16:27] really positive things from metrics [16:30] um mostly it's the fact that when we [16:33] measure something it can prove us wrong [16:35] and we can learn counterintuitive [16:37] things. Lots of good stuff, but there's [16:40] also a flip side to it that I mean a [16:43] good example would be we've kind of [16:44] established quite firmly about [16:46] throughput and stability in the Dora um [16:49] research. So, we know that deploying [16:51] more often, having shorter change lead [16:53] times, these are all good things, but [16:56] also we've all used software on our [16:58] phones and when it nags us every day to [17:01] say there's a new version of this app, [17:03] it's actually irritating, right? So um [17:06] if we just take the metrics in [17:09] isolation, we want to deploy as often as [17:11] possible, but we're not listening to the [17:14] users and being close to the users. We [17:16] can create situations like that where [17:19] like I have some apps that I've [17:21] seriously considered just deleting the [17:22] app because I'm so fed up with being [17:24] told every day there's an update for it. [17:26] Um it's it's a burden on me as a user. [17:30] So metrics don't always give you the [17:31] complete picture. And you know I think [17:33] there are some famous quotes from past [17:36] like uh leaders you know if you go into [17:38] that Ducker and Deming space there's a [17:40] huge amount of stuff that we can't [17:42] effectively measure and human judgment [17:45] is needed. So don't just depend on [17:47] metrics. [17:51] >> I was actually going to say something [17:53] very similar that like these days it [17:55] feels like you can come up with metrics [17:57] for anything but you need to be careful [17:59] that you're measuring the right thing. [18:01] And I think you know what you said Steve [18:03] about like the end user experience and [18:05] you know your phone nagging you like oh [18:07] yeah I've I've done the same thing and I [18:09] think we can get so bogged down in the [18:11] metrics to steal your sandwich metaphor [18:13] we can be like okay we got to make sure [18:14] that we are getting the cheese on that [18:16] sandwich really really fast and you you [18:18] get so focused on that particular metric [18:21] that you forget that you're supposed to [18:23] be serving burritos instead of [18:24] sandwiches and it's yeah people can get [18:28] really kind of rabbit holed into let's [18:30] make sure that the metrics look good [18:32] because oftentimes, you know, that can [18:34] be how your team gets headcount or you [18:36] get a bigger budget is if you meet your [18:38] metrics. So, you got to make sure you're [18:39] meeting your metrics. But ultimately, [18:42] we're not doing what we're doing in [18:43] order to make the numbers on a graph do [18:46] a certain thing. We are trying to solve [18:48] problems for people. And so, it's really [18:50] important to keep that in mind and not [18:52] just focus on the metrics. [18:57] Uh I I would add that an example by [19:02] setting a strict goal like commit and go [19:05] to stage environment and x minutes is [19:08] not a good met at all because when [19:12] people try to achieve those goals or [19:14] pursuing it is like by maybe removing [19:18] test and reducing validation to reach [19:20] the number and I think it's not just [19:24] what we want to We want to go to safely [19:27] depo deployment. We get to we um have [19:33] this healthy loop inside of all the the [19:37] circle. [19:39] >> Yeah, that's such a good point. Like if [19:41] you if you want that deployment to be [19:42] super fast, just delete all the tests. [19:44] Hooray. Yeah, it's you can see people [19:46] doing that. And if you actually think [19:48] about the outcomes you're expecting, you [19:51] kind of yes, you still want to try and [19:54] get feedback to the developer really [19:55] quickly so they know if they've broken [19:57] something, they're almost like, I know [19:58] which line of code I've just done um [20:00] that's caused this, so I know exactly [20:02] where to look. Pinpointing cost of zero. [20:05] Um uh but if you have longer running [20:07] tests, that's fine, too. Right? After [20:10] you've given some feedback to the [20:11] developers, you carry on doing more [20:12] tests. You run some tests in parallel. [20:15] But yeah, you wouldn't want to reduce [20:17] the the valuable steps, you know, and [20:20] what happens if we if we do that, we [20:22] remove the test automation, but now we [20:24] need to have a actual like manual [20:26] testing phase that's much longer. [20:29] >> So, you know, we're kind of we're [20:30] robbing ourselves, I think, if we do [20:32] that. [20:34] Yeah, [20:38] I'm I'm worried that if we continue on [20:39] metrics, we may spend all day speaking [20:41] about metrics, but I I think I totally [20:44] agree with what you're saying. You know [20:45] like if I give an example I used to be [20:47] in in the platform team where the goal [20:49] was to reduce 30% of the you know like [20:52] the the request requests were from [20:55] people using the platform and I used to [20:57] joke around that like a very good way [20:58] would be to delete the slack channel [21:00] where we would get the request you know [21:02] it will certainly improve the metrics [21:04] but it wouldn't achieve the goal and the [21:06] reason is because some some metrics are [21:10] lagging you're going to find out only [21:12] when uh you know when things are [21:14] happening. that it's a bit too late to [21:16] navigate if you're doing the right [21:17] direction or not and it is extremely [21:19] hard to find a leading metrics that help [21:22] you figure out if you're going [21:23] direction. So, but I'm going to pause on [21:25] metric because I think we may going to [21:27] get too excited about it and move to [21:29] something else. I wanted to touch base [21:30] with you all. So, I want to go back to [21:33] what something we said before that is [21:35] like another systemic issue. Why do [21:37] teams tolerate broken feedback loops for [21:40] so long? Why don't they fix them [21:42] earlier? Uh there is certainly a problem [21:44] that I'm feeling at times myself. So [21:47] Francine, what do you think? What what's [21:49] your take? [21:52] >> Um I I think that people get used to [21:55] them. It's something always have been [21:58] slow. Uh people stop questioning and [22:02] just become how things works. [22:05] And I I think there's also some kind of [22:08] roadmap pressure because making [22:12] improvements in pipelines and looking [22:14] for the feedback loop feels like doing [22:17] some internal work and the internal work [22:21] will not give the vis the visible [22:24] progress in the road map. So it will get [22:28] the prior the prioritize and [22:31] uh as a like a platform engineer I would [22:34] say that another factor is that uh [22:38] there's not clear ownership about it and [22:42] this usually cross boundaries like the [22:45] code the platform the infrastructure so [22:48] there's no single team that feels fully [22:51] responsible for it and what some some [22:55] thing that I've seen is when there is [23:00] these bunch of pull requests that are [23:03] opened by bots like these bots that will [23:07] bump library versions or do some little [23:10] things in the code and sometimes this [23:14] request will fail in with a llinter [23:18] issue and in that point there's this [23:22] kind of uh manually approach that a [23:25] person need to pull the branch locally, [23:28] fix the link error, commit, push, ask [23:32] for a pull request, review again and [23:34] then merge the pair uh merge the PR. I I [23:37] think this is repetitive and is this is [23:40] something that I in my experience I pass [23:43] it through and what we've uh solve it or [23:48] improve the the feedback loop in this [23:51] case because the system was given the [23:53] feedback the link failed. So something [23:55] that we've I I've done is instead of [24:00] keeping the manual process, we introduce [24:02] a small automation. When a comment with [24:07] uh a comment in a p request with a [24:09] command that will uh trigger the [24:12] pipeline will run the linked fix [24:15] command, commit the change and update [24:18] the p request automatically. So the [24:20] person could always review again the [24:23] request and merge it without the need of [24:27] uh doing the manual work uh again. So I [24:31] I think this is uh people will tolerate [24:34] these broken feedback feedback loops [24:37] because the system is still functions [24:40] but is just inefficient. [24:45] Is it like similar to the principle of [24:47] like make it easy to do the right next [24:50] action or something like that? Is that [24:52] is that on this page? Yeah, [24:54] >> there's an element of that I think. [24:55] Yeah, I kind of feel that humans have [24:57] this built-in mechanism like when we put [24:59] our clothes on, our brain knows to stop [25:02] telling us we're wearing clothes [25:04] um so that we don't go around all day [25:06] thinking, "Oh, I'm wearing clothes." [25:07] Right? It's like they're rubbing on us [25:09] and they're there. But our brain just [25:10] goes, "You don't need to know about [25:12] that." And I think that same mechanism [25:14] can work when we've got flaky tests or [25:17] the build's taking a bit too long or [25:19] humans are having to intervene in what [25:20] should be automated pull requests. So I [25:22] think there's there's something very [25:24] like uh bronze age about that part of [25:27] our brain. [25:31] I think also there can be just competing [25:34] priorities where the team just has too [25:37] much to do or you know it's not the [25:41] shiny feature work that is going to you [25:43] know look really good on a slide deck [25:45] somewhere and it's just hard to justify [25:49] yeah we're going to you know stop [25:50] building new stuff for customers and go [25:53] fix the builds like that's that's not [25:55] glamorous and shiny. So people even if [25:58] they want to do it they aren't [26:00] necessarily getting you know the [26:01] approval from from managers or above you [26:03] know that that level up higher uh to to [26:06] be able to do those things. I think one [26:08] thing that can be really interesting is, [26:10] you know, ask the engineers on the team [26:13] like if you could wave a magic wand and [26:15] fix one thing, what would it be? And [26:17] then really listen to those results [26:19] because I think a lot of times people do [26:21] like yet there is definitely a [26:23] normalization of deviance of just like [26:25] we we forget the friction when it's [26:27] there every day. But I think there's [26:29] also a lot that it's like, oh, I would [26:30] love to do that. Maybe I'll wait until, [26:33] you know, nobody's looking and then [26:34] just, you know, the day when everyone [26:36] like working in Europe with a Europe [26:39] with an American team. You know, the day [26:41] of Thanksgiving and the day after is [26:42] when I'm like, I'm going to go fix the [26:44] builds because nobody's around to tell [26:45] me not to. [26:49] >> Yeah, [26:49] >> I love that. Maybe we need to celebrate [26:51] those things more as well. Like we don't [26:53] pop champagne corks when we make builds [26:56] better and when we make deployments [26:58] better and when our observability is [26:59] telling us things before a customer [27:01] knows about it. We need to start popping [27:03] champagne for those as well. [27:06] Um I'll tell you something that I [27:08] started doing in January. Maybe it's [27:09] useful to others. Um I started doing [27:12] like a a tech lens for my department [27:15] where I on purpose mostly asked to do to [27:19] showcase things that basically like I I [27:22] care about. So at the moment I care [27:24] about improve operability. I care about [27:26] like you know systemic architecture [27:28] changes allow us to be more efficient. [27:30] So on so basically I I cherry pick and I [27:33] say teams I want to just talk about this [27:34] stuff. It doesn't mean the other stuff [27:36] it doesn't it doesn't have to be [27:38] listened to right just that there are [27:39] other avenue where we we have a lot of [27:41] champagne pop paying for those stuff. So [27:44] it's a way to normalize that there is [27:46] that there is some ground work that [27:48] maybe it's not celebrated enough but [27:50] when it does it like it pay off a huge [27:53] amount. So that's something that is [27:55] working well well in case it's something [27:57] that you're interested in implementing. [27:59] Um but yeah let let me R given that you [28:04] you kind of mentioned it so I wanted to [28:05] go a bit deeper on this one. Uh Ren [28:08] sorry um if you are an engineering [28:13] manager and you know your build as low [28:15] but you have a road map pressure how do [28:17] you actually justify investing in [28:19] pipeline improvements? So you you [28:21] mentioned it you mentioned the pain but [28:23] tell me more about what you think about [28:24] the solution could be. Yeah, I think in [28:27] a lot of bit I mean it'll depend based [28:29] on what your org values but you know if [28:32] we talk about metrics again a lot of a [28:35] lot of companies love metrics and [28:36] especially if you're trying to justify [28:38] this to the higherups they care about [28:40] the the metrics that have dollar signs [28:42] in front of them. So, if you're able to [28:44] put together something that says like, [28:46] "Hey, we've spent, you know, x number of [28:49] hours waiting for slow builds or this [28:52] this number of hours rerunning the flaky [28:55] tests or, you know, when production went [28:59] down for this amount of time because we [29:01] didn't have a proper staging environment [29:03] set up and we couldn't test things [29:04] properly, you know, that lost us this [29:07] amount of money." And if you can put [29:08] dollar signs in front of it, then you [29:11] can usually, you know, get the buy in to [29:14] to focus on those sorts of things. Um, [29:17] if you're not on a platform team, it can [29:19] also help to go ask the platform team [29:21] for help because like being on a [29:24] platform team, we have the ability to [29:26] kind of scale that out. So, we're not [29:28] just fixing one team's pipelines, but, [29:30] you know, we could make changes that can [29:32] scale up everyone's, you know, make [29:34] everyone's builds faster, more reliable. [29:37] So finding ways to leverage that scale [29:39] where you can say okay the you know the [29:41] platform team can help you know fix [29:43] these CI problems which will enable the [29:46] feature teams to do X Y and Z then you [29:48] can kind of talk about those trade-offs [29:50] that way. [29:52] >> I also think as a profession we need to [29:55] bear in mind that we are professionals [29:58] doing a difficult job and in a lot of [30:02] cases this this question comes up for [30:04] things like TDD as well. It's like we we [30:06] haven't got time and we need to ask [30:07] permission to get time to do it. But by [30:11] doing it, you're freeing up your own [30:12] time. So you can make a small investment [30:15] and then use the time you win back from [30:17] that small investment to reinvest again [30:19] and fixing things like tests, uh, build [30:23] servers, deployment, automation, [30:24] observability, [30:26] all pay you back in time. So you put an [30:29] hour in, you get time back, you put that [30:31] time in, and you get increasing amounts [30:33] of time to improve this stuff. Um, and [30:36] if my plumber came to me and said, "Oh, [30:38] I wanted to use copper pipes here, um, [30:41] because if I use these plastic pipes, [30:44] they're not as good and the joints [30:45] aren't as strong." I'd be like, "Why are [30:46] you telling me this?" I'm like, "I I [30:48] trust you. I've paid you to come and do [30:50] my plumbing. I would like a quality [30:52] outcome. You don't have to ask me [30:54] permission on what pieces to use. And if [30:58] there's a regulation that says you have [30:59] to put in these isolated valves, I'd [31:01] expect you to just do it." Like don't [31:02] come and say, "Oh, should I follow the [31:04] regulation and put in isolator valves?" [31:06] It's like your job as a professional is [31:08] to know all of that and just make it [31:09] happen. [31:15] >> Yeah. I think this is an end thing like [31:18] uh Steve ran both of you said need to [31:20] play at the same time. I really like [31:22] what Ran said because I think it's [31:24] something that we don't do enough of. We [31:26] don't justify I think even like the [31:29] existence of our teams. you know, I've [31:31] been in a situation where the existence [31:33] of the platform team wasn't well, you [31:35] know, communicated and then there was a [31:37] layoff and we lay off a large amount of [31:40] platform team and surprise the speed or [31:43] flow, whatever you want to call it, it [31:45] went down. So, it's kind of like I think [31:47] um it resonated a lot with me. Thank you [31:49] for sharing Paul. Um [31:53] maybe [31:57] let me think about where we want to go. [31:59] I'm I'm keen to understand um what [32:02] systemic shift open to you all uh not [32:06] just tooling tweaks uh will materially [32:09] improve feedback speed today it's kind [32:12] of like you know if you think globally [32:14] not necessarily tooling what will help [32:18] Steve you want to give it a go and then [32:20] we see [32:20] >> yeah I can jump in I think um one of the [32:24] the big untapped um resources that we [32:27] all have available to us is um user [32:31] feedback. Um uh I've worked in a lot of [32:34] organizations where as a developer um [32:38] the idea of you speaking directly to the [32:40] people using the software was almost [32:41] like what are you thinking? Go and sit [32:44] in your basement man you know. Um, but [32:47] we now know that like connecting that up [32:51] so developers are sitting with real [32:54] users of the software, seeing how they [32:56] use it, having conversations [32:58] makes you a high performing product [33:00] team. Um, and even if you're slightly [33:03] slower in terms of deployment frequency [33:05] and lead times, that user centricity [33:08] actually brings your performance up to [33:10] like the highest levels. So um I think [33:14] at the moment there's so much [33:16] conversation about flatline speed that [33:19] this user centricity part is missing and [33:22] that's that's like a gold mine of [33:25] untapped resource that we should be [33:27] looking at. [33:31] I think that's really true and [33:32] especially when you have you know [33:35] internal teams whose customers are other [33:38] engineering teams like the platform [33:40] teams or the dev tools team or you know [33:42] those sort of internal teams like we've [33:44] been focused a lot at at Mapbox on how [33:47] to improve communication you know [33:49] between platform and our customers and I [33:52] think really you know when you're [33:54] talking about internal teams there's no [33:55] excuse because these people are you know [33:57] at the ne next desk or the Slack channel [34:00] over. Um, so finding ways to to get that [34:05] feedback from from your customers, like [34:08] you can't not do it. Um, the other thing [34:11] I think is interesting is kind of [34:14] cultural feedback loops where we're not [34:17] just thinking about how do we make our, [34:20] you know, CI pipelines better, but how [34:22] do we make our overall work processes [34:24] better. So doing stuff like [34:26] retrospectives with the team to just [34:28] talk about like hey how how is how is [34:31] work going for everyone? Are there [34:33] things that are going well? Are there [34:34] things that are going not well? What [34:36] could we be doing better to work [34:38] together? You know especially when you [34:41] have these, you know, oh it's all about [34:43] the DevOps. How do we make sure that [34:45] we're working together as effectively as [34:48] possible and tightening up all of those [34:50] feedback loops, whether they're [34:52] technical or not, [34:54] >> and putting the action items from your [34:56] retro onto the same task board that you [34:59] put all of the other work. That's that's [35:00] probably the biggest mistake I've made [35:02] throughout my whole career is forgetting [35:03] to actually write that stuff down to [35:06] make space for it. [35:09] And building on that that Ren said about [35:12] listen to the our customers like we are [35:16] from platform and listen to our [35:18] customers. I I think doing these surveys [35:21] is really insightful when we are [35:24] developing or maybe tweaking those [35:27] little things that makes the daily [35:30] working more uh joyable. I would say [35:33] it's not like a pressure in the day that [35:37] I know this is is low again. So I can [35:41] tweak this and this we can have this [35:44] feedback from our customers. I would say [35:47] that we should treat as we as platform [35:51] engineers, we should treat the platform [35:53] as a product and our customer as our [35:56] real customers doing this uh surveys [35:59] this and uh have this actional [36:04] actionable items [36:07] >> and to all of my friends in the platform [36:08] engineering world that does mean you [36:10] can't just decide what is good for [36:12] people like I know that like platform [36:15] engineers are often the people that have [36:17] the most experience with these pieces, [36:18] which is why they're in platform [36:20] engineering now. But yeah, don't don't [36:23] be overconfident that you know what is [36:26] needed. You have to connect it back to [36:28] real needs. [36:30] Yeah, I mean uh another [36:34] picy take here is that it's natural that [36:37] you know easily um if the platform [36:40] wasn't mandatory you you know you will [36:43] see exactly if actually what you built [36:45] fit what people want to use it for [36:47] because people just doing themselves but [36:50] it's challenging where the only avenue [36:52] to do some stuff is by their platform [36:54] because then it doesn't force the people [36:56] who are building the tool to think hard [36:58] about like what's the right abstraction [37:00] level, what is the right way to on board [37:02] people and so on. So I think it's a yeah [37:05] it's a it's it's an interesting [37:07] challenge for large and cost like heavy [37:10] platform where actually everybody doing [37:12] what they want is is not an easy option. [37:15] So I appreciate that and um Ren I have [37:20] another actually another another thing I [37:23] wanted to run by you that is um [37:27] I design loops um so that I don't just [37:33] so that basically remaining sustainable [37:35] and human for teams um I think it's also [37:38] like sometime we go you know maybe we [37:42] talk a bit when we talk about metric but [37:43] sometime we go a bit over board. So, um, [37:46] what's your take? [37:49] >> Yeah, I think it's [37:51] like I' I've said this before, I'll say [37:53] it again, I'll say it forever. We have [37:55] to remember that we are solving problems [37:57] for people, not for computers, not for [38:01] metrics. So, thinking about what are we [38:03] actually trying to accomplish? Like I I [38:06] sometimes I have to, you know, be a [38:08] customer who writes into customer [38:10] service myself and you can kind of tell [38:12] as a customer when somebody is being [38:14] judged on the metric of how quickly they [38:16] close the tickets because that's when [38:18] you get the copypaste response that [38:20] doesn't answer your question, but it [38:22] came in 30 seconds and you can you can [38:24] tell what that metric is and you can [38:26] tell that that is not a happy person who [38:28] is probably having a great time at their [38:31] jobs. And I think yeah, it's important [38:33] to remember the the human side of [38:35] things. So for managers, keep keep tabs [38:39] on how your how your teams are doing, [38:42] how you know fulfilled people are listen [38:46] when people are saying like this isn't [38:48] working, this isn't sustainable, and [38:50] find ways to to try and make those [38:53] improvements as much as you can. I think [38:55] celebrating the wins like we we talked [38:58] about the champagne earlier and [39:00] especially for internal teams like [39:02] platform teams that are often just seen [39:04] as a cost center that's like what do we [39:06] even have them for we could just you [39:08] know outsource that to chat GPT right so [39:10] being able to find ways to share like [39:13] hey these are the wins like these are [39:15] this is what we did for our customers [39:17] you know on the on the other dev teams [39:20] this is what we did that's valuable so [39:22] you can share that within the org which [39:24] not only helps you celebrate the wins, [39:26] but also kind of show shows the value [39:29] that you bring to the org that might [39:30] otherwise just be invisible because I [39:32] think we tend to have one of those jobs [39:34] where if you're doing it really well, [39:36] nobody notices that you exist. [39:40] >> Yeah. And you can be really lightweight [39:43] in checking in on um how developers [39:46] feel. Uh, I'm pretty sure I'm stealing [39:47] this idea from Jim Benson where he said [39:50] each card that you complete, just draw a [39:52] face on it, like a smiley face or an [39:54] unhappy face or a me face. And you can [39:57] get a really good temperature check of [39:59] how the team is doing. And as leaders, [40:03] we have to be super careful that we [40:05] don't explain away when people aren't [40:07] feeling great. It can be tempting to [40:09] kind of put a narrative around it or [40:11] say, "Oh, that that person is quite [40:13] negative or whatever, but actually [40:16] they're sending signals. The reason [40:17] you're having to explain it away is [40:18] because they're almost like waving and [40:21] saying help." Um, so dig deeper and try [40:24] not to just rationalize why people [40:27] aren't happy. Go and find out what you [40:29] could do to improve that situation. [40:34] >> Yeah, makes sense. And uh I I I I [40:38] started seeing a couple of question [40:39] coming from the audience. So what I [40:41] would like to do is to try to get the [40:43] last one of our question that I have and [40:46] then I wouldn't mind to respond to one [40:48] and then we leave the other for slack if [40:51] it's okay with you all. Should we do it? [40:53] >> Yeah. [40:53] >> Um okay Francine um I I would love your [40:58] take. So uh let's say if you are we are [41:01] advising a leadership team today uh what [41:03] leading indicator would you track to [41:05] know whether your feedback loops are [41:07] healthy? I know we talk about like the [41:09] you know the peril of measuring but it's [41:12] like if if you had to do it what what [41:14] would you what would you measure? [41:17] Um, I would focus on lead indicators [41:21] that show how fast and how safely the [41:25] system learns. From, for example, a [41:28] quantitative perspective, I track things [41:32] like build time, test retries, flake [41:35] tests, and automated rollbacks. If those [41:39] start increasing, uh, I I I say it [41:42] feedback is slowing down or becoming [41:45] less reliable. But if only the numbers [41:50] are not enough, I would pay attention to [41:54] the qualitative signals. Uh if engineers [41:57] start normalizing friction as we talk [42:00] about or creating workarounds that uh uh [42:05] an early uh warning signing that [42:08] something isn't healthy. So I would pay [42:10] attention in these kind of indicators. [42:15] >> Thank you. [42:17] Thank you. So let me get like uh a [42:21] question that I quite like uh and then [42:23] we get the other one in the in the [42:25] channel. Um the question is uh there are [42:28] so many feedback loops. Are we building [42:30] the right things? Are we building it [42:32] right? Are we test are the test passing? [42:34] How is the runtime performance? How is [42:36] the team morale and so on? Since it's [42:38] almost impossible to do them all at the [42:39] same time, which feedback loop do you [42:41] leave for last and why? Um so obviously [42:45] like [42:46] Hi. [42:47] >> Yeah, that's fascinating. I think I [42:50] always look at these feedback loops like [42:51] an onion. So the developer's inner loop [42:54] is in the middle, which is when you're [42:55] you write one small line, one small [42:58] change, and you want to make sure that [43:00] the the software still builds, the test [43:02] still pass. There's like a little loop [43:04] there, and that's helping to guide you, [43:06] right? And you're probably adding tests [43:08] as you go as part of that because you're [43:09] asking for a change. So you're either [43:11] editing a test or adding a new test. Um, [43:14] and then you kind of like go out a [43:15] little bit and now there is someone that [43:17] isn't you now giving you feedback on it. [43:20] Maybe there's a code review and they're [43:22] pointing out that you've opened a [43:23] transaction way too early and that's [43:25] going to cause performance problems. And [43:28] then there's another layer outside of [43:29] that where maybe someone does some [43:31] exploratory testing and they're like, [43:33] well, this the way that this form works [43:35] is confusing. I think users are going to [43:37] trip up on this. And so it kind of keeps [43:39] going outwards all the way to even past [43:42] the user feedback. Users can say I love [43:44] using the app. And then there's an even [43:46] wider piece of feedback which is are [43:49] they paying an amount of money which [43:51] allows you to stay in business. Right? [43:54] So which is like the ultimate feedback [43:56] is what does the bank balance look like [43:58] for your organization? Um and how's that [44:01] trending? So that's kind of how we prove [44:04] end to end that we are doing something [44:06] valuable in the world. [44:09] >> Thank you Steve. We had run out of time [44:11] so I need to close but is let's follow [44:14] up on Slack uh because I think there are [44:16] a lot to say about the topic. So I [44:19] wanted to thank you. We don't didn't [44:20] just talk about like sandwiches and [44:22] tacos. Uh we we really talk about like [44:25] feal loops and how to tackle them. I [44:28] wanted to thank you Ren Steven Francine. [44:31] It was generally like a very rich [44:33] discussion and it really brought me joy [44:35] and a lot of very good ideas. So, thank [44:37] you so much. [44:38] >> Thank you very much as well. [44:39] >> Thank you. [44:40] >> Thank you.  📄 **Full text:** ![[Full Document Contents/Articles/Why Good Teams Rely on Bad Workarounds|Why Good Teams Rely on Bad Workarounds, YouTube]] ## Highlights - painful or even when some of the metrics are misleading if we talk about customer feedback uh then team tend to adapt. they kind of like they adjust to the fact that there are things that are a bit Sorry for for the sword. Uh and those adaption are often rational. It kind of makes sense. Uh but I think in the long term they are going to cost the company quite a lot of money. So today I would us to explore where feedback loops normally break down. um ([View Highlight](https://read.readwise.io/read/01kmv7a9ckycycjt3pddpb4df6)) - Tags: 👻 ai highlighted - figure out what's going on. but just these little annoyances that you never really notice until somebody new joins your team and then you have to explain them to somebody and you're like that's kind of embarrassing. Why didn't we fix that? Um, so I think it can be really important, especially when we're talking about these internal feedback loops, to just think about ways of capturing that information. And one thing that I like to do is like do a team hack week every so often where you you know prioritize fixing those things that nobody's had time time or priority to fix before. ([View Highlight](https://read.readwise.io/read/01kmv7a9dasqwvmy68je8wmern)) - Tags: 👻 ai highlighted - this healthy loop inside of all the the circle. >> Yeah, that's such a good point. Like if you if you want that deployment to be super fast, just delete all the tests. Hooray. Yeah, it's you can see people doing that. And if you actually think about the outcomes you're expecting, you kind of yes, you still want to try and get feedback to the developer really quickly so they know if they've broken something, they're almost like, I know which line of code I've just done um ([View Highlight](https://read.readwise.io/read/01kmv7a9h2srt7h7mzy22fmmxr)) - Tags: 👻 ai highlighted - with you all. So, I want to go back to what something we said before that is like another systemic issue. Why do teams tolerate broken feedback loops for so long? Why don't they fix them earlier? Uh there is certainly a problem that I'm feeling at times myself. So Francine, what do you think? What what's your take? >> Um I I think that people get used to them. It's something always have been slow. Uh people stop questioning and ([View Highlight](https://read.readwise.io/read/01kmv7a9k9sb1h4qajbz8ffva0)) - Tags: 👻 ai highlighted - just become how things works. And I I think there's also some kind of roadmap pressure because making improvements in pipelines and looking for the feedback loop feels like doing some internal work and the internal work will not give the vis the visible progress in the road map. So it will get the prior the prioritize and ([View Highlight](https://read.readwise.io/read/01kmv7a9mww0seytpd2kvrfa8y)) - Tags: 👻 ai highlighted - over. Um, so finding ways to to get that feedback from from your customers, like you can't not do it. Um, the other thing I think is interesting is kind of cultural feedback loops where we're not just thinking about how do we make our, you know, CI pipelines better, but how do we make our overall work processes better. So doing stuff like retrospectives with the team to just talk about like hey how how is how is ([View Highlight](https://read.readwise.io/read/01kmv7a9whfedvgvz98zrxzh3h)) - Tags: 👻 ai highlighted - the right things? Are we building it right? Are we test are the test passing? How is the runtime performance? How is the team morale and so on? Since it's almost impossible to do them all at the same time, which feedback loop do you leave for last and why? Um so obviously like Hi. >> Yeah, that's fascinating. I think I always look at these feedback loops like an onion. So the developer's inner loop is in the middle, which is when you're you write one small line, one small change, and you want to make sure that ([View Highlight](https://read.readwise.io/read/01kmv7aa7pec4527cevm6srgmx)) - Tags: 👻 ai highlighted ## New Highlights added 2026-03-28 at 7:22 PM - painful or even when some of the metrics are misleading if we talk about customer feedback uh then team tend to adapt. they kind of like they adjust to the fact that there are things that are a bit Sorry for for the sword. Uh and those adaption are often rational. It kind of makes sense. Uh but I think in the long term they are going to cost the company quite a lot of money. So today I would us to explore where feedback loops normally break down. um ([View Highlight](https://read.readwise.io/read/01kmv7a9ckycycjt3pddpb4df6)) - Tags: 👻 ai highlighted - figure out what's going on. but just these little annoyances that you never really notice until somebody new joins your team and then you have to explain them to somebody and you're like that's kind of embarrassing. Why didn't we fix that? Um, so I think it can be really important, especially when we're talking about these internal feedback loops, to just think about ways of capturing that information. And one thing that I like to do is like do a team hack week every so often where you you know prioritize fixing those things that nobody's had time time or priority to fix before. ([View Highlight](https://read.readwise.io/read/01kmv7a9dasqwvmy68je8wmern)) - Tags: 👻 ai highlighted - this healthy loop inside of all the the circle. >> Yeah, that's such a good point. Like if you if you want that deployment to be super fast, just delete all the tests. Hooray. Yeah, it's you can see people doing that. And if you actually think about the outcomes you're expecting, you kind of yes, you still want to try and get feedback to the developer really quickly so they know if they've broken something, they're almost like, I know which line of code I've just done um ([View Highlight](https://read.readwise.io/read/01kmv7a9h2srt7h7mzy22fmmxr)) - Tags: 👻 ai highlighted - with you all. So, I want to go back to what something we said before that is like another systemic issue. Why do teams tolerate broken feedback loops for so long? Why don't they fix them earlier? Uh there is certainly a problem that I'm feeling at times myself. So Francine, what do you think? What what's your take? >> Um I I think that people get used to them. It's something always have been slow. Uh people stop questioning and ([View Highlight](https://read.readwise.io/read/01kmv7a9k9sb1h4qajbz8ffva0)) - Tags: 👻 ai highlighted - just become how things works. And I I think there's also some kind of roadmap pressure because making improvements in pipelines and looking for the feedback loop feels like doing some internal work and the internal work will not give the vis the visible progress in the road map. So it will get the prior the prioritize and ([View Highlight](https://read.readwise.io/read/01kmv7a9mww0seytpd2kvrfa8y)) - Tags: 👻 ai highlighted - over. Um, so finding ways to to get that feedback from from your customers, like you can't not do it. Um, the other thing I think is interesting is kind of cultural feedback loops where we're not just thinking about how do we make our, you know, CI pipelines better, but how do we make our overall work processes better. So doing stuff like retrospectives with the team to just talk about like hey how how is how is ([View Highlight](https://read.readwise.io/read/01kmv7a9whfedvgvz98zrxzh3h)) - Tags: 👻 ai highlighted - the right things? Are we building it right? Are we test are the test passing? How is the runtime performance? How is the team morale and so on? Since it's almost impossible to do them all at the same time, which feedback loop do you leave for last and why? Um so obviously like Hi. >> Yeah, that's fascinating. I think I always look at these feedback loops like an onion. So the developer's inner loop is in the middle, which is when you're you write one small line, one small change, and you want to make sure that ([View Highlight](https://read.readwise.io/read/01kmv7aa7pec4527cevm6srgmx)) - Tags: 👻 ai highlighted ## New Highlights added 2026-03-28 at 8:22 PM - painful or even when some of the metrics are misleading if we talk about customer feedback uh then team tend to adapt. they kind of like they adjust to the fact that there are things that are a bit Sorry for for the sword. Uh and those adaption are often rational. It kind of makes sense. Uh but I think in the long term they are going to cost the company quite a lot of money. So today I would us to explore where feedback loops normally break down. um ([View Highlight](https://read.readwise.io/read/01kmv7a9ckycycjt3pddpb4df6)) - Tags: 👻 ai highlighted - figure out what's going on. but just these little annoyances that you never really notice until somebody new joins your team and then you have to explain them to somebody and you're like that's kind of embarrassing. Why didn't we fix that? Um, so I think it can be really important, especially when we're talking about these internal feedback loops, to just think about ways of capturing that information. And one thing that I like to do is like do a team hack week every so often where you you know prioritize fixing those things that nobody's had time time or priority to fix before. ([View Highlight](https://read.readwise.io/read/01kmv7a9dasqwvmy68je8wmern)) - Tags: 👻 ai highlighted - this healthy loop inside of all the the circle. >> Yeah, that's such a good point. Like if you if you want that deployment to be super fast, just delete all the tests. Hooray. Yeah, it's you can see people doing that. And if you actually think about the outcomes you're expecting, you kind of yes, you still want to try and get feedback to the developer really quickly so they know if they've broken something, they're almost like, I know which line of code I've just done um ([View Highlight](https://read.readwise.io/read/01kmv7a9h2srt7h7mzy22fmmxr)) - Tags: 👻 ai highlighted - with you all. So, I want to go back to what something we said before that is like another systemic issue. Why do teams tolerate broken feedback loops for so long? Why don't they fix them earlier? Uh there is certainly a problem that I'm feeling at times myself. So Francine, what do you think? What what's your take? >> Um I I think that people get used to them. It's something always have been slow. Uh people stop questioning and ([View Highlight](https://read.readwise.io/read/01kmv7a9k9sb1h4qajbz8ffva0)) - Tags: 👻 ai highlighted - just become how things works. And I I think there's also some kind of roadmap pressure because making improvements in pipelines and looking for the feedback loop feels like doing some internal work and the internal work will not give the vis the visible progress in the road map. So it will get the prior the prioritize and ([View Highlight](https://read.readwise.io/read/01kmv7a9mww0seytpd2kvrfa8y)) - Tags: 👻 ai highlighted - over. Um, so finding ways to to get that feedback from from your customers, like you can't not do it. Um, the other thing I think is interesting is kind of cultural feedback loops where we're not just thinking about how do we make our, you know, CI pipelines better, but how do we make our overall work processes better. So doing stuff like retrospectives with the team to just talk about like hey how how is how is ([View Highlight](https://read.readwise.io/read/01kmv7a9whfedvgvz98zrxzh3h)) - Tags: 👻 ai highlighted - the right things? Are we building it right? Are we test are the test passing? How is the runtime performance? How is the team morale and so on? Since it's almost impossible to do them all at the same time, which feedback loop do you leave for last and why? Um so obviously like Hi. >> Yeah, that's fascinating. I think I always look at these feedback loops like an onion. So the developer's inner loop is in the middle, which is when you're you write one small line, one small change, and you want to make sure that ([View Highlight](https://read.readwise.io/read/01kmv7aa7pec4527cevm6srgmx)) - Tags: 👻 ai highlighted