How guardrails enable you to work at AI speed

How guardrails enable you to work at AI speed 🔗 **Source:** [How guardrails enable you to work at AI speed](https://www.youtube.com/watch?v=C_m5OEg42zw) > [!note]+ Document Notes > tags: [ai-guardrails, progressive-delivery, developer-experience, ai-safety, devops] > author: Lead Dev > source: YouTube > reference: https://www.youtube.com/watch?v=C_m5OEg42zw > date created: 2024-12-19 11:30 > > # 🏷️ Title: How guardrails enable you to work at AI speed >  ## 🧰 Action Inbox - [H1] Implement contextual intelligence in AI pipelines — understand why failures occur, not just that they failed #ai-pipelines - [H2] Create AI Bill of Materials (AIBOM) for model tracking — know the source and origin of all AI models in use #security - [H3] Establish behavioral anomaly detection for AI systems — monitor for drops in model accuracy or drift #monitoring ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [Progressive Delivery Book](https://progressivedelivery.com) - [Lead Dev Slack Community](https://leaddev.com/slack) ## 📝 Summary - **What happened**: Panel discussion on implementing guardrails for AI-enabled development without sacrificing speed - **Why it matters**: Organizations are struggling to balance AI adoption velocity with safety, security, and compliance requirements - **Key stakeholders**: Engineering leaders, DevOps teams, platform engineers, and developers adopting AI tools - **Risks and tradeoffs**: Speed vs. safety, automation vs. human oversight, experimentation vs. compliance - **Bottom line takeaway**: Guardrails should enable safe experimentation at AI speed through progressive delivery, not block innovation through rigid gates ## 📖 Main Content Progressive delivery provides a framework for safely adopting AI at scale by implementing guardrails that protect both users and systems while enabling rapid iteration. Unlike traditional gate-based approaches, guardrails allow controlled experimentation with defined blast radius limits. Key principles include understanding your risk profile, implementing tiered deployment strategies, and maintaining human oversight where it matters most. Organizations must balance autonomous team empowerment with necessary compliance requirements, especially in regulated industries like fintech. The four-pillar framework for AI guardrails includes: contextual intelligence (understanding why failures occur), automatic verification (AI Bill of Materials tracking), behavioral anomaly detection (monitoring for model drift), and continuous learning loops that adapt guardrails over time. ## 🗒️ Notes #### ⏱️ [00:00](https://youtu.be/C_m5OEg42zw?t=0) — [H1] Introduction to AI Speed vs Safety Challenge - Key idea: Moving fast with AI requires guardrails, not just speed alone - Evidence: "Just fast is a recipe for disaster. That frankly, I haven't seen in the headlines yet, but I'm waiting for that to happen" #### ⏱️ [05:23](https://youtu.be/C_m5OEg42zw?t=323) — [H2] Progressive Delivery Definition and Philosophy - Key idea: Progressive delivery breaks down barriers between DevOps and users through cooperative releases - Evidence: "Release not a fiat that we impose on people but a cooperative event that we are sharing" #### ⏱️ [08:15](https://youtu.be/C_m5OEg42zw?t=495) — [H2] Wise's Approach: Golden Paths and Service Tiers - Key idea: Mature service tiering with mandatory canary deployments for critical services - Evidence: "All of tiers one and two have to use automatic canary deployments and analysis and there's no escaping that" #### ⏱️ [12:30](https://youtu.be/C_m5OEg42zw?t=750) — [H2] Compliance Challenges in Regulated Industries - Key idea: DORA regulations require mapping all services to critical business functions - Evidence: "If one system goes down we know the critical function it supports and then the blast radius of what else that could affect" #### ⏱️ [16:45](https://youtu.be/C_m5OEg42zw?t=1005) — [H2] AI Development Complexity Trade-offs - Key idea: AI increases code contributions but creates integration bottlenecks - Evidence: "6% increase in code contributions but an 8% increase in complexity of integrating the AI driven development" #### ⏱️ [20:12](https://youtu.be/C_m5OEg42zw?t=1212) — [H1] Four-Pillar Framework for AI Guardrails - Key idea: Comprehensive framework addressing AI-specific risks and monitoring needs - Evidence: "Contextual intelligence, automatic verification, behavioral based anomaly detection, and continuous learning" #### ⏱️ [23:40](https://youtu.be/C_m5OEg42zw?t=1420) — [H2] AI Bill of Materials (AIBOM) - Key idea: Need to track AI model origins like software dependencies - Evidence: "Previously we had our SBOM software bill of materials but now we have AI BOM" #### ⏱️ [26:55](https://youtu.be/C_m5OEg42zw?t=1615) — [H2] Risk Assessment First Principle - Key idea: Understanding risk profile determines appropriate AI adoption strategy - Evidence: "The first question has to be about risk. How much risk my actions as a developer will incur" #### ⏱️ [31:20](https://youtu.be/C_m5OEg42zw?t=1880) — [H2] Monitoring AI Systems: "Is It Running?" vs "Is It Right?" - Key idea: Traditional monitoring insufficient for AI systems requiring correctness validation - Evidence: "Normally until now we've asked the question of is it running, but now we also have to ask is it right?" #### ⏱️ [34:10](https://youtu.be/C_m5OEg42zw?t=2050) — [H2] AI Failure Pattern Detection - Key idea: Using smaller models to evaluate larger models for common failure modes - Evidence: "Prompt injection, topic drift, PII leakage, toxicity, bias, hallucinations" #### ⏱️ [37:25](https://youtu.be/C_m5OEg42zw?t=2245) — [H2] Human-Centric AI Philosophy - Key idea: Maintaining human agency while leveraging AI as amplification tool - Evidence: "Pilots over passengers - we should always be the pilot with the AI, not the passenger to it" #### ⏱️ [40:30](https://youtu.be/C_m5OEg42zw?t=2430) — [H2] Intelligence Over AI Focus - Key idea: Emphasize making systems smarter regardless of underlying technology - Evidence: "Move the mindset from AI to intelligence, intelligence is much more general" #### ⏱️ [42:50](https://youtu.be/C_m5OEg42zw?t=2570) — [H2] User Acceptance and Change Resistance - Key idea: Technical capability doesn't guarantee user adoption - Evidence: "Microsoft has 500 million users who have PCs capable of upgrading to Windows 11 who are refusing" #### ⏱️ [45:15](https://youtu.be/C_m5OEg42zw?t=2715) — [H2] Documentation as AI Sweet Spot - Key idea: AI-augmented documentation offers high value with manageable risk - Evidence: "Supervised AI documentation...probably going to be one of the more powerful things that we pull out" ## 📚 References - [Progressive Delivery Book](https://progressivedelivery.com/) - [Lead Dev Community](https://leaddev.com/) - [Harness Platform](https://harness.io/) - [Spotify Backstage](https://backstage.io/) - [DORA Research](https://dora.dev/) ## 🗣️ Paraphrase - Progressive delivery enables safe AI adoption by replacing rigid approval gates with flexible guardrails that allow controlled experimentation - The four-pillar AI guardrails framework provides contextual intelligence, automatic verification, behavioral monitoring, and continuous learning to manage AI-specific risks - Organizations must balance development velocity with appropriate risk management based on their regulatory environment and customer impact potential ## 📰 Newsletter **AI Guardrails: Moving Fast While Staying Safe** The AI development boom presents a classic trade-off: speed versus safety. A recent Lead Dev panel featuring experts from Wise, Coralogix, and Harness explored how teams can implement guardrails that enable rapid AI adoption without compromising security or compliance. The key insight? Replace traditional approval gates with intelligent guardrails that allow controlled experimentation. Heidi Waterhouse, co-author of the Progressive Delivery book, emphasized that guardrails protect not just your team but the entire ecosystem. **Four Essential Pillars for AI Guardrails:** - **Contextual Intelligence**: Understand why failures occur, not just that they happened - **Automatic Verification**: Implement AI Bill of Materials (AIBOM) to track model origins - **Behavioral Monitoring**: Detect when model accuracy drops or behavior drifts - **Continuous Learning**: Adapt guardrails as AI systems evolve **Practical Applications:** - Use service tiers to determine deployment requirements (critical services need canary deployments) - Monitor for AI-specific failure patterns like prompt injection and hallucination - Implement "pilots over passengers" philosophy - keep humans in control - Focus on high-value, lower-risk applications like AI-augmented documentation The panel stressed that effective AI guardrails are living systems that evolve with your technology stack, not static rules that become obsolete. Resources: [Progressive Delivery Book](https://progressivedelivery.com/), [Lead Dev Community](https://leaddev.com/slack), [DORA Research](https://dora.dev/) ## ✅ Action Items #### DevRel - Developer advocacy: Create content on AI guardrails best practices, host workshops on progressive AI adoption - Education: Develop training materials on the four-pillar AI framework, risk assessment workshops - Developer experience: Document AI tooling policies, create templates for AI project evaluation - Developer marketing: Promote human-centric AI philosophy, showcase safe AI adoption case studies - Community management: Facilitate discussions on AI governance, moderate AI policy feedback #### OSPO - Policy & Governance: Develop AI usage policies, create AI Bill of Materials standards - License Compliance: Review AI model licensing, audit open-source AI dependencies - Security Management: Implement prompt injection scanning, establish AI vulnerability reporting - Software Inventory: Track AI models like software dependencies, maintain AIBOM - Developer Support: Provide AI governance training, create evaluation frameworks - Community Engagement: Participate in AI safety standards development - Strategic Alignment: Define AI adoption KPIs, measure guardrail effectiveness #### Community - Maintainers: Create AI contribution guidelines, establish model evaluation criteria - Contributors: Onboard contributors to AI safety practices, provide AI development training - Sponsors: Fund AI safety research, support responsible AI development - Ecosystem: Coordinate on AI standards, share behavioral anomaly detection patterns #### WordPress - Core: Evaluate AI integration opportunities, implement content generation safeguards - Core Performance: Monitor AI-generated code performance impact, profile AI features - Core AI: Establish AI safety committee, develop evaluation datasets - Community: Create AI usage guidelines for contributors, moderate AI-generated content - Training: Develop AI safety curricula, create responsible AI workshops - Documentation: Use AI to enhance documentation discoverability while maintaining accuracy - Security: Scan for AI-generated vulnerabilities, monitor for prompt injection attempts #### The WPCC - Programs: Fund AI safety research, provide grants for responsible AI tooling - Projects: Support AI governance documentation, fund safety evaluation tools - Funding: Allocate budget for AI risk assessment, support model auditing initiatives #### FAIR - Technical independence: Ensure AI systems maintain community governance - Policies: Establish transparent AI decision-making processes - Governance artifacts: Create AI usage charters, document model selection criteria ## 🧪 Hypotheses - Organizations implementing the four-pillar AI guardrails framework will achieve faster AI adoption with fewer production incidents - Service tier-based deployment requirements reduce AI-related customer impact while maintaining development velocity - Human-centric AI approaches ("pilots over passengers") lead to better long-term adoption and fewer resistance issues - Progressive delivery principles applied to AI features result in higher user acceptance rates than traditional rollouts ### 📊 Methodology - Comparative analysis of incident rates before and after guardrail implementation across different service tiers - Survey research on developer productivity and satisfaction with AI tools under different governance models - Case study analysis of organizations using progressive delivery for AI feature rollouts - Longitudinal study of user acceptance patterns for AI-enhanced versus traditional software releases ### ❓ Questions - How do we measure the effectiveness of behavioral anomaly detection in preventing AI system failures? - What metrics best predict user acceptance of AI-enhanced features before full rollout? - How should AI Bill of Materials standards evolve to address emerging model architectures and deployment patterns? ### 🔮 Future Research - Development of standardized AI risk assessment frameworks across different industries - Investigation of automated guardrail adaptation based on system behavior and user feedback - Exploration of collaborative AI governance models for open-source projects ### 💡 Implications - Practice: Development teams need new skills in AI risk assessment and behavioral monitoring - Policy: Regulatory frameworks must evolve to address AI-specific compliance requirements - Security: Traditional security scanning insufficient for AI systems requiring new detection capabilities - Education: Engineering curricula should include AI governance and progressive delivery principles - Funding: Organizations need budget allocation for AI monitoring infrastructure and governance tooling ### 🧠 Learning Styles Assessment Reading level estimate: Flesch-Kincaid Grade Level 12 #### 👀 Visual learners - Present: Service tier diagrams, four-pillar framework illustration - Improve: AI pipeline flow diagrams, risk assessment matrices, guardrail architecture visuals #### 🔊 Aural learners - Present: Full 45-minute panel discussion with expert insights - Improve: Audio summary of key takeaways, podcast-style follow-up discussions #### 🗣️ Verbal learners - Present: Clear definitions of progressive delivery, comprehensive analogies (guardrails vs gates) - Improve: Industry-specific terminology explanations, more real-world examples #### 🧩 Kinesthetic learners - Present: Practical implementation examples from Wise and other organizations - Improve: Step-by-step guardrail implementation checklists, hands-on workshops #### 🧠 Logical learners - Present: Structured four-pillar framework, clear cause-effect relationships - Improve: Decision trees for risk assessment, quantified metrics for guardrail success #### 🧑‍🤝‍🧑 Social learners - Present: Panel format with multiple expert perspectives, community discussion elements - Improve: Facilitated peer learning sessions, collaborative guardrail design workshops #### 🧘‍♀️ Solitary learners - Present: Individual reflection prompts, self-assessment opportunities - Improve: Personal risk assessment worksheets, independent study materials #### 🌿 Naturalistic learners - Present: Ecosystem thinking about AI impact, organic evolution of guardrails - Improve: Environmental sustainability considerations for AI systems, lifecycle thinking ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading structure, descriptive section titles - Improve: More detailed alt text for framework diagrams, ARIA landmarks for navigation #### Keyboard navigation - Present: Standard video controls, accessible timestamp links - Improve: Skip-to-content links, keyboard shortcuts for section navigation #### Contrast and readability - Present: High contrast text, readable font sizes - Improve: Ensure 4.5:1 contrast ratio for all text elements, optimize line height #### Text resizing and reflow - Present: Responsive design supporting zoom up to 200% - Improve: Test reflow at higher zoom levels, prevent horizontal scrolling #### Media and captions - Present: Full --- ## Full Transcript [00:12] Hi everyone. I'm really excited to be [00:14] here today. Uh thank you so much for [00:17] both those that are joining us live and [00:19] to my esteemed panelists and those for [00:22] listening the recording for discussion [00:24] on how guardrails enable you to work at [00:26] AI speed. We all want this AI speed, but [00:30] we can't just be fast. Just fast is a [00:33] recipe for disaster. That frankly, I [00:36] haven't seen in the headlines yet, but [00:38] I'm waiting for that to happen. It's [00:39] like I'm Yeah. Well, Heidi said I think [00:43] Heidi's face like, well, there's been [00:44] some headlines, but I think the worst [00:46] has not happened yet. Hopefully, we can [00:49] get guards rules in place to keep that [00:52] worst from happening, at least affecting [00:53] us in our own blast radius. Uh this is a [00:57] lead dev webinar created in partnership [00:59] with harness. It's going to last about [01:01] 40 45 minutes after which all of us five [01:05] will head over to the lead dev slack to [01:08] answer more of your questions in the [01:10] hashtag panel and Q&A channel to [01:13] continue the discussion. If you aren't [01:15] already a member, I'm going to add it [01:17] into the chat now. But it's just a very [01:20] fun way. It's a very good community and [01:23] supportive community. So it's a good way [01:26] to just continue the discussion. So not [01:29] just in this channel, but it's just a [01:32] really good channel of engineering [01:34] leaders. For those joining us live, feel [01:36] free to share your questions here as [01:38] well. So we'll continue the conversation [01:41] in the Slack, but if you have [01:43] interesting questions or things to share [01:44] throughout, please check the Q&A. So [01:47] let's get started with some [01:48] introductions. I'm Jennifer Riggins. I'm [01:50] a technology journalist. And today I'll [01:52] be joined by or I am joined by Heidi [01:55] Waterhouse, co-author of the progressive [01:57] delivery book, Build the Right Thing for [02:00] the Right People at the Right Time. [02:02] Annie Freeman, developer advocate at [02:04] Corore Logix. Rick Kle, developer [02:07] experience lead at Wise, and Dwan Ahmed, [02:10] principal developer advocate at harness. [02:13] So if you I think that's a great order [02:16] to start in if you remember. So how it [02:18] went. Heidi, Annie, Rick, then Zoan, if [02:20] you want to go around and introduce [02:22] yourselves more fluidly, that would be [02:25] great. Thank you so much. [02:26] >> Sure. I'm Heidi Waterhouse. I'm [02:29] currently independent and I just [02:31] finished co-authoring this book on [02:34] progressive delivery. So, that's been [02:36] really exciting. I also have a book [02:38] about documentation for developers. And [02:42] in my experience, I've been doing a lot [02:45] of work on how it is we get things out [02:47] to people in an effective way. [02:51] >> Okay, great. Annie. [02:54] >> Hi, I'm Annie. I'm a developer advocate [02:56] at Corel Logix. I've recently moved over [02:58] to the UK from New Zealand actually [03:00] where I used to be a software engineer [03:01] at zero doing like backend and [03:03] infrastructure work but now at [03:05] Crowologics I'm all focused on the world [03:08] of observability and uh really [03:11] interested in the AI aspect of that and [03:13] how we observe and use AI uh in that [03:16] world as well and then alongside that [03:18] I'm also a big uh green software [03:20] advocate so how we can reduce the [03:22] environmental impact of technology [03:25] especially uh in with the world of AI. [03:29] >> So important. Absolutely. And that is [03:32] part of measurement that shouldn't be [03:34] neglected from any measurements that [03:36] we're doing. Rick. [03:38] >> Hey, I'm Rick. I'm the developer [03:40] experience lead at Wise. Uh I have kind [03:44] of lots of different uh missions. We are [03:47] really focused on engineering [03:48] productivity. [03:50] uh as a larger fintech now we our [03:55] mission is to move at the speed of a [03:57] startup but with all the security and [03:59] compliance of a bank. Um, and you can [04:02] imagine when you're really focused on [04:05] engineering productivity and we've just [04:07] entered this big hype world of AI that [04:10] is all over mine and Wise's radar right [04:14] now of how to empower engineers and like [04:16] we've said not break anything at the [04:19] same time. [04:21] >> Okay, Dwan, please. [04:23] >> Hey, I'm Dwan Ahmed. I'm joining from [04:25] the beautiful East Coast, New Brunswick. [04:27] It's snowy here and I work at a company [04:30] called Harness. We help engineering [04:32] teams to ship code faster, safer and [04:36] smarter. I work as a principal developer [04:38] advocate. That means part of my work is [04:40] I join these awesome panels with really [04:42] smart people talking about technology. [04:45] But the more important part is what I [04:47] hear from the community, I bring it back [04:49] to the engineering team so that we can [04:51] improve on the technology we all use to [04:53] ship software. [04:56] >> Okay, great. So, we chose a topic called [04:59] guardrails, which of course I think [05:02] Netflix famously coined the term [05:04] guardrails, not gates. [05:07] I don't drive, but I know what a [05:08] guardrail is in that sense. But I think [05:11] Heidi, you're probably best since you [05:13] literally wrote the book on it to define [05:15] this concept of progressive delivery to [05:17] kick us off, please. [05:19] >> Sure. So the way that we've been [05:22] defining progressive delivery is that it [05:26] we need a way to break down the barrier [05:29] between DevOps and users between making [05:32] the software and the people who are [05:34] consuming it. And the way we do that is [05:38] to make release not a fiat that we [05:42] impose on people but a cooperative event [05:46] that we are sharing. So people have more [05:50] ability to pull or reject what it is [05:52] that we're offering. When we think about [05:55] progressive delivery, we need to build [05:57] teams that have the capacity to listen [05:59] to how users are experiencing software [06:02] and um what they actually need from it. [06:05] And so all of this wraps into the idea [06:08] of guardrails as like what is it that we [06:11] can do to make it safe for people to [06:14] adopt so that they are not skeptical or [06:17] worried or frightened about adoption [06:19] because like it's not irrational to say [06:22] like I'm not installing v0.1 of [06:25] something. [06:27] And that's kind of what we're asking [06:29] people to do with a lot of AI powered [06:31] stuff. It's not deterministic. We don't [06:34] know how it's going to work. So, we need [06:36] to build in safeguards that keep us from [06:42] from hurting anybody. And guardrails [06:44] aren't just about preserving the people [06:46] in the car that hits the guardrail. [06:48] They're also about preserving everybody [06:49] else on the road. [06:52] >> Oh, dark but valid in the icy season. [06:58] That's very interesting. And yeah, uh, [07:00] another point of progressive delivery I [07:03] think involves that like blast radius. [07:05] So maybe it's progressively releasing [07:08] things because you may need to roll it [07:11] back or things like that. So it's very [07:14] interesting because we have [07:17] Annie works for more of a fast growing [07:19] company and Ricky I know you said like [07:22] the speed of a startup [07:25] but the security and compliance of a [07:26] bank but I assume you're probably not as [07:30] fast moving at least externally because [07:33] of the inherent nature that you are [07:35] still one of the widest spread banks [07:38] like I'm an expat so everyone I know [07:41] uses it to transfer across different [07:43] borders uses wise. So [07:47] >> what are the what does progressive [07:48] delivery mean in the context of each of [07:51] your business? Ricky, do you want to [07:52] talk from the more well regulated? Not [07:55] that every industry shouldn't be [07:57] regulated in some way, but you know it's [08:00] a bit different when you're in banking. [08:02] >> So the major only gate that we have is [08:06] like the four eyes coding rule. like no [08:09] code can enter production without two [08:12] humans having looked and approved it. So [08:17] outside of that we really try to only [08:20] have guard rails and there's several [08:24] ways we've gone about doing that. Um I [08:28] don't want to I guess spoil them all [08:30] with other questions coming but some of [08:32] like our key ones are golden paths. We [08:36] have something we call wise Java [08:38] platform. We are big advocates and users [08:42] of Spotify's backstage which we then use [08:46] their scaffolder system which allows us [08:49] to have templated service creation. [08:53] So it gives a norm to engineers which [08:56] they can be familiar with and help them [08:58] to drive quicker. My teams really look [09:01] for a way to like reduce the toil so [09:07] engineers can really use their powerful [09:09] human brains to work on great product [09:12] features and problems and not how to [09:15] maintain and update Spring Boot [09:17] versions. [09:19] >> So just for those listening if you're [09:21] not familiar with this term golden paths [09:24] I we were literally talking offline but [09:26] I promise I've been about wicked but I [09:28] promise I've been using this. you've [09:29] missed before the wicked of it all. Uh I [09:32] look at it as in some I think it should [09:34] been called the yellow brick road if [09:35] that's not TMs. Uh because the idea is [09:39] that on golden path it's the easiest way [09:41] to get from point A to point B but if [09:44] you go off you might get tacked by [09:46] winged monkeys. You might fall asleep in [09:48] a poppy field [09:51] but you can. So guard rails [09:56] unless you're jumping them it's much [09:59] harder to go off. So do you have a [10:01] differentiation between at wise or [10:04] anywhere between the golden paths that [10:07] is like your platform engineering the [10:08] easiest way to do something and the [10:10] guard rails where you cannot really [10:12] unless you're actively aggressively [10:14] trying to not do something. So the only [10:18] thing where you're really locked down [10:20] like like a lot of companies we started [10:22] off in the 2010s are this concept of [10:25] autonomous teams and of course as we've [10:27] grown and become more regulated it's now [10:30] like empowered teams like and in [10:32] platform we of course never want to get [10:34] involved in like the what is being [10:37] built. We are not the experts there but [10:39] if we want to move into worlds like [10:41] fleet management then we need to have [10:44] some kind of more consistent engineering [10:47] path. The major thing we've had to make [10:50] decisions on especially with AI and the [10:53] speed we want to move is that we have [10:56] like a mature service tiering system. So [10:59] all of tiers one and two have to use [11:03] automatic canary deployments and [11:06] analysis and there's no escaping that. [11:10] So that is a hard rule. But if you are [11:13] our lower peers that have either none or [11:16] little customer impact then you are [11:18] still free to deploy in whichever manner [11:22] best helps you be productive and [11:24] deliver. [11:25] >> Okay. I have more wait yeah sorry [11:27] >> I have a question about that. Please, [11:30] >> how do you know for sure that something [11:32] doesn't have customer impact? Because I [11:34] think one of the interesting things [11:36] about platform is that you can have [11:39] customer impact without knowing that you [11:41] have that dependency. [11:43] >> So there are very new ways we do it and [11:47] more older ways. The older ways we were [11:50] referencing this through if when you are [11:52] part of wise Java platform we have very [11:54] good dependency management which we can [11:57] visualize and is cataloged within [11:58] backstage. The actual more modern way is [12:01] I don't know if it's affected anyone on [12:03] the panel but the DORA digital [12:06] operations resilience act means that we [12:08] have all of WIS's critical functions [12:11] mapped and all the services in that are [12:14] run at Wise are mapped to those critical [12:16] functions. So although the mapping is [12:19] perfect and but what you get from that [12:22] is if one system goes down we know the [12:24] critical function it supports and then [12:26] the blast radius of what else that could [12:30] affect. [12:32] >> Okay. Yeah. Uh just because we are in [12:34] tech there's another dora that famously [12:36] had a Google. Yes. This is a very [12:39] important regulation that not only if [12:42] that I've shared in the chat and we'll [12:44] share in the slack but not only must you [12:48] comply if you are a European business [12:50] but if you are doing businesses in with [12:54] European people [12:55] >> it is it's definitely confusing when [12:57] you're in the DevX world because of of [13:00] course as you said Dora productivity [13:02] metrics [13:03] >> um and sharing acronyms. [13:06] So, something that came up in a [13:09] conversation I had last week with [13:12] McDonald's and JP Morgan Chase is that [13:18] Chase can't like legally can't offer [13:21] different things to different people [13:24] because that would break compliance. So, [13:26] you could do like a different colored [13:27] button or something, but if you're going [13:30] to roll out a new feature, you can't [13:32] just to your partners or something. you [13:34] have to roll it out. So [13:38] because legally you can't offer banking [13:40] services to some people if not others. [13:43] Wise is more global. So maybe are you [13:46] doing that based on different I don't [13:49] know countries or jurisdictions or how [13:52] can you do canary deployment or is it [13:55] only in your platform engineering [13:57] internal developer customers? [13:59] >> No, it's it's fully external. I I can't [14:03] talk of course for sure of all the [14:05] different things like restricting our [14:08] releases. Um we have a system in house [14:12] called feature service which does our [14:14] feature flagging for it's not really [14:17] granular like we can really target but [14:20] allows us to switch on features with our [14:22] app our back end our front-end services [14:25] but the way the canary analysis works [14:28] um I don't know the when it comes down [14:34] to what we offer but wise already offers [14:36] different products to different [14:37] jurisdictions we have our assets [14:39] product. I hope you're all aware. Of [14:41] course, if you're in the UK, you can use [14:43] that, but that's not available in all of [14:45] our countries either. So, I haven't got [14:48] a extremely clear answer there, but [14:51] Okay. [14:52] >> Very interesting. Annie, you're on [14:56] I I know you have some sort of [14:58] compliance and restriction, but you are [15:00] a rapid moving like scale up in a scale [15:03] up stage. So, how is it different? How [15:05] can you be potentially more experimental [15:07] in a controlled way? [15:09] >> Yeah, definitely. So, at Corlogix, it's [15:13] a full stack observability platform. So, [15:15] I think from our perspective, we're [15:18] doing whatever we can to enable people [15:21] to develop with AI to produce uh use AI [15:25] in production. But how do you do that uh [15:28] in a way like how can we enable these [15:30] safeguards to be put in place uh like [15:33] enabling constraints for example so that [15:35] you can do that effectively. Uh I was [15:37] reading an article recently that was [15:40] explaining how people were I think it [15:45] said that developers had found that [15:48] using AI to help with development was [15:51] led to about a 6% increase in code [15:54] contributions but an 8% increase in uh [15:57] like complexity of integrating the uh AI [16:02] driven development into it. So it's sort [16:04] of creating a bottleneck there. you've [16:06] you've got this increase in efficiency, [16:08] but you don't have necessarily it's very [16:10] localized. It's not uh you know working [16:13] completely. Uh so how do we enable [16:17] people to to hopefully remove slowly [16:20] that that bottleneck and how do you also [16:24] uh instrument AI models themselves and [16:27] deploy those with confidence? [16:30] >> Absolutely. No, and it's so important. [16:34] Uh I definitely think I Dan you had an [16:39] analog anal you made an analogy [16:43] that AI is a bullet train going past the [16:48] rail like it's automating insecurity at [16:52] scale. [16:53] How well yeah we know just like Annie [16:56] said about it's increasing this [16:58] complexity. [17:00] We don't know what's being put into what [17:02] I mean open AAI is objectively a large [17:05] research project where we are the [17:08] unwilling recipients of it. So we don't [17:11] fully know what's going out there and we [17:14] can only do so much with the what we're [17:17] using what we're building but then [17:19] there's also everything we're [17:20] integrating with any data coming in and [17:22] out. So this is all while test coverage [17:25] has never been enough. So what guard [17:29] rails do we need to set up earlier in [17:32] the face of this more code than ever [17:34] that also maybe not at wise but well no [17:39] they just had four eyes so two [17:41] developers have to have seen but looked [17:43] at the code that doesn't mean [17:45] necessarily created the code maybe just [17:48] looked at code that they pair programmed [17:49] with an AI [17:51] >> so what Dan do you think we can add [17:54] sooner [17:55] >> yeah I think like there's a lot of doom [17:57] and gloom right? What things can go [17:59] wrong? But first, I want to take a [18:00] moment to appreciate the work we have [18:02] done in the last few decades. So, we [18:04] have done a ton of work improving the [18:06] SDLC, shift left mindset, incredible [18:10] amount of automation. All of those [18:11] things worked. So, now we're super [18:13] proud, patting ourselves in the back [18:14] saying that now we have a solid uh I [18:18] guess delivery procedure. But most of [18:21] our delivery pipelines and everything, [18:23] we had humans in the loop. Humans used [18:25] to write code. Humans used to test. [18:26] Humans were the ones clicking the [18:28] approve. Yes. Now deploy to production. [18:30] Now what happens with AI is we are [18:32] getting a massive amount of code [18:34] generated by AI. We have uh AI making [18:37] decisions where our pipelines are [18:39] heading. So this speed uh is nice but [18:43] it's also creating a huge amount of [18:45] risk. The other part is it's creating a [18:48] different sort of thread landscape where [18:50] before you had like SQL injection but [18:52] now you have prompt injection. Now you [18:54] have model tampering. So all these [18:56] different challenges are coming uh and [18:58] as uh leaders we need to either evaluate [19:02] if our team has enough capacity to build [19:05] a platform that has guardils in place or [19:08] we use something an existing platform [19:10] something like harness that has these [19:12] these guards in place and I I know we [19:14] talked last week and I was talking about [19:16] the four framework I I proposed uh uh [19:19] that that we can use. One of them is [19:22] contextual intelligence. That means the [19:24] pipeline has to know the context. Not [19:26] just something failed but why it failed. [19:29] Automatic verification. Uh previously we [19:32] had our sbomb software bill of materials [19:34] but now we have AI bomb. So the AI bill [19:36] of materials where the models are coming [19:37] from. So automatic verifications to know [19:39] the source of origin. Then the [19:42] behavioral based anomaly detection that [19:44] means uh if I now have a pipeline that [19:46] is doing an sentiment analysis model and [19:49] let's say the accuracy of that model [19:51] dropped from 90% to 40%. Of course all [19:54] the checks are still passing but how can [19:57] I understand that behavior changed and [20:00] based on that the fourth pillar is [20:01] continuous learning. So the guardrails [20:03] and I I totally agree to what Heidi [20:06] Annie and Rick were mentioning about the [20:07] guardrails for their specific industries [20:10] but I'd also like to add is this [20:11] guardrail will be a living guardrail. [20:14] It's not you set up one guardrilla and [20:15] that's done for the next five years. You [20:18] have to constantly adopt that guardrail [20:20] as well. So it's a moving guardrail. [20:24] >> Okay. Yeah. Absolutely. Because we're [20:26] definitely in a very moving time period. [20:28] So I just want to reveal the four review [20:30] the four pillars again just because that [20:32] was a lot of really good information. [20:34] We've shared it in the chat and we'll [20:35] share in slack on slack as well. Lead [20:38] dev slack really good place to be. Uh [20:40] pillar one is contextual intelligence. [20:43] Then we have automatic verification, [20:45] behaviorbased [20:47] anomaly detection and continuous [20:49] learning loops which I think feeds very [20:51] back well into the continuous feedback [20:53] loops that DevOps promised and may or [20:56] may not have delivered. [20:58] So these are things that we have to work [21:02] on and we're [21:04] we're what three and a half years into [21:06] the AI of it all or the AI of it all in [21:08] our software development not the AI of [21:10] it all for data scientists etc. So as we [21:14] do that, there's a shocking low amount [21:18] of user policies and things since [21:20] everyone's using it, not just [21:22] developers, but we're kind of focus on [21:23] developers. Heidi, like what questions [21:27] should we be asking as we're examining [21:32] our own AI policies and our own [21:34] guardrails? [21:35] >> I think the first question has to be [21:37] about risk. like I [21:42] I want to have a really clear idea on [21:47] how much risk my my actions as a [21:50] developer will incur. So like there's a [21:53] bunch of places where you can't use AI [21:56] because there are regulations on it and [21:59] some of that is finance and a lot of it [22:02] is life critical programming like the [22:05] the OSS that run planes are nothing like [22:07] the OSS we deal with. Um and the medical [22:11] device people are enormous nerds about [22:16] everything. Um and we appreciate that. [22:19] So they already know that they're a [22:22] critical path and they take a different [22:24] degree of care, but all of us are a [22:27] critical path for something. Um, you [22:30] know, when [22:33] when we get an attack that takes down [22:35] our library, it's a critical path for [22:38] somebody. They're not going to die, but [22:40] they're going to be deeply [22:42] inconvenienced and maybe lose their job. [22:45] Uh so when we are thinking about what [22:48] kind of thing we need to think about [22:50] first we have to think about our risk [22:52] profile and um the more I consider this [22:57] the more I'm like we're all a little [22:59] highrung about people's ability to [23:01] gamble online um or or play Candy Crush [23:06] or even stream their movies. Like we we [23:11] do a lot of S sur talking about uptime [23:13] for something that [23:15] in the end is only existential to our [23:18] jobs and not to other humans. [23:20] >> Yeah. [23:20] >> I hope you're enjoying this video, but [23:22] did you know lead dev has a lot more to [23:25] offer? [23:25] >> When I think about lead dev, I think [23:27] about bricks and hearts. Bricks are [23:30] those foundational things that you need [23:31] to know and lead dev provides that [23:34] through webinars, through videos, [23:36] through cutting edge research. But [23:38] hearts are the things that you need to [23:40] glue yourself to the people at work and [23:43] lead does that through conferences and [23:46] meetups. [23:46] >> Make sure you check out leadv.com. [23:54] >> And I think that's really useful to [23:56] consider when we're talking about adding [23:58] AI, like does this have to be [24:00] deterministic or is there room for some [24:04] experimentation? Is there room for this [24:06] to go wrong? [24:09] Okay, Dwan. Yeah, this is Is there room [24:12] to this to go wrong? Yes. Being the [24:14] answer. [24:16] >> No, I I love that I I love that angle [24:18] about uh risk. And one thing I'd like to [24:21] like uh discuss with the panel is how [24:24] much risk should we allow to an [24:25] individual developer versus how much the [24:28] guardrail should be strong. And what I [24:30] mean by that is let's say a developer [24:32] unintentionally if they're now [24:33] committing AWS API keys with their code [24:37] um do they even need to uh be exposed to [24:41] that level of risk or there should there [24:42] be something like a gitlink scanner or [24:44] or a secret scanner that will just even [24:47] prevent them from committing if they are [24:49] now trying to deploy um some model to [24:53] the production um that is not safe um [24:56] will there the guardrail uh allow allow [24:59] them to to push through or uh so in that [25:02] sense I'm asking how much the risk limit [25:06] we can lower for individual engineering [25:08] teams. [25:11] >> Yeah. And I think that's a really good [25:13] point is that like on a team you need to [25:16] be able to be very autonomous and the [25:19] output of the team is a promise [25:23] and at what level do you need to say [25:25] like the buck stops here and the promise [25:28] must be fulfilled. [25:32] And just in any way when you're [25:35] encouraging this time of extreme [25:36] experimentation [25:39] uh make sure that it they're safe to [25:41] fail, they're safe to experiment. [25:43] There's psychological safety issues in [25:45] there as well that we especially in this [25:47] time of layoffs where you have to make [25:50] it totally like you have to make people [25:54] feel safe in their jobs so they can be [25:56] creative in a way. So Annie, we've been [26:00] talking a lot about the the before and [26:02] the during now once it's out in the [26:05] wild, we'll say the code. How do you [26:08] know what are we looking for into what [26:10] the AI systems are doing? How do we gain [26:12] visibility in there throughout the life [26:14] cycle? [26:16] >> Yeah, great question. And I think a good [26:17] way to like sum this up is like normally [26:21] until now it's we've asked the question [26:23] of is it running, but now we also have [26:26] to ask is it right? Uh and so there's so [26:29] much that's going on behind the scenes [26:31] that you not might not necessarily have [26:33] visibility of. So you don't have a [26:35] traditional stack trace. You don't [26:37] necessarily have a line of code. Uh you [26:41] need to be some other way of of [26:43] understanding this. Uh what you can do [26:45] is uh instrument with open telemetry. We [26:48] have a already a lot of things already [26:50] in place for tracing, stack tracing, u [26:53] distributed tracing especially for [26:55] agentic systems uh that you can put in [26:58] place and that can lead to enabling you [27:01] to use other small language models or [27:04] evaluators to go and look at uh common [27:07] AI failure patterns. So things like you [27:10] were saying before prompt injection or [27:12] uh topic drift, PII, uh leakage, uh you [27:16] can look for things like toxicity, bias, [27:18] uh hallucinations. Uh and I think using [27:23] AI in that way as well is incredibly [27:26] useful to keep up with the amount of [27:28] scale and speed of producing AI in [27:32] production as well. [27:34] >> I heard an acronym in there and I love [27:36] to identify acronyms. P AI Sorry, my [27:40] apologies. Personally identifiable [27:41] information. [27:42] >> Oh, PI. Okay, fair enough. Yes, that's a [27:45] very good one for us all to know though. [27:48] Yes, absolutely. And [27:52] do we are we how do we know if tools are [27:55] even able to keep up with the scale? [27:57] Like, are we measuring [28:00] are our systems going to crumble because [28:02] we're putting we're moving too fast? [28:05] We're putting too much stuff out there. [28:08] Yeah, that's definitely a a potential [28:10] issue with uh monitoring AI systems [28:13] because we already generate huge huge [28:15] amounts of telemetry data, but now if [28:18] you're monitoring AI systems, you're [28:20] generating even more. For example, you [28:22] can you most likely want to uh be [28:25] reading like full chat histories that [28:28] your users are having so you can check [28:30] if there's any of these issues cropping [28:32] up. Uh and so that's a huge amount of [28:34] data that you're now having to analyze [28:37] and store uh and do something with. So [28:41] that yeah, we need to think about the [28:42] scale of that as well. [28:46] >> Absolutely. What else do we all I'm [28:49] going to pop Cornet out to the floor. [28:52] What else do we think the questions we [28:54] should be asking of our systems of our [28:56] leadership [28:58] in this age of AI? How do we set guard [29:01] rails without asking questions first? [29:04] >> I think it's important for leadership to [29:08] set the strategy, the policy, the [29:12] culture of how the tools should be used [29:17] and what fits within your organization. [29:21] Like if you are a [29:25] money-m [29:27] startup looking to please VCs then you [29:30] might want to work in another way. If [29:31] you're a much more human centric [29:33] organization [29:35] which I believe wise is then we always [29:38] believe humans are the superpower and AI [29:42] is a tool to help us produce things and [29:45] our thinking. So that culture and that [29:48] needs to come from leadership and then [29:50] of course engineers do get excited. So [29:53] that's when our security squad will make [29:55] security policies to try and make sure [29:58] we stay within safety. I think the big [30:01] thing uh like Annie said for us is [30:03] there's nothing more important than our [30:05] customer data. So we need to be our most [30:09] careful around those areas. Of course, [30:12] there's great potential where AI can [30:14] help with fraud analysis with [30:16] transferring money and things like that, [30:18] but you know, we need to be really [30:20] careful with how that we use that. We [30:22] have platform teams that build what we [30:25] would call the LLM gateway where so [30:28] everything will go through that. That's [30:29] to help with our observability and [30:31] tracing everything that's going on [30:34] to try and use these things in a great [30:37] way to empower rather than control. Um, [30:41] I didn't coin the phrase, but I heard [30:43] this great phrase which was pilots over [30:45] passengers [30:47] and that we should always wise be the [30:50] pilot with the AI, not the passenger to [30:53] it. uh which helps with understanding [30:57] what's going on more than anything. [31:00] >> So in that euphemism, what is AI on the [31:03] airplane? [31:05] >> Well, the I don't know about [31:09] co-pilot, not to sell uh GitHubs and my [31:13] coding agent, but the the co-pilot. I I [31:17] guess for me, and you used the phrase, I [31:20] can't remember if it was during or just [31:21] before the call, which was they're like [31:23] your pair programming partner. That's [31:26] what as as the agent is for me, at least [31:29] on a local because I I guess I'm more [31:31] interested from my perspective of what [31:33] our engineers are doing with the tools [31:35] rather than them in the system with [31:36] customers like and and they're your pair [31:39] programming partner. Like no different [31:41] to how you would talk and interact with [31:44] a human and come up with great ideas. [31:46] you would go around in circles [31:48] sometimes, sometimes you'd go in a [31:50] tangent and then reverse that. And those [31:52] patterns are still the same. Um, sadly [31:55] with a little less human connection, but [31:57] also your pair programming partner can [31:59] be an expert in SQL even if a regular [32:02] human might only know about [32:06] >> Fair enough. Okay, Dan. [32:10] Yeah, I to answer the question about [32:12] what the team should be asking about uh [32:14] what to do in in this AI era. It reminds [32:17] me the the first time people were [32:19] adopting Kubernetes uh they were [32:21] thinking that do I need Kubernetes to [32:23] deploy my personal blog? So like just [32:25] Kubernetes on everything. So I think [32:28] it's a similar time frame where AI on [32:30] everything. So if we move the mindset [32:32] from AI to intelligence, intelligence is [32:35] is much general. So it doesn't have to [32:37] be just AI. So for example, if you're [32:40] running 12,000 unit tests, you can have [32:43] an intelligent CI pipeline uh that looks [32:45] at the call graphs and only run the test [32:49] based on the the code changes uh and the [32:51] dependency. So you might run 500 tests, [32:54] let's say for for one run, and then [32:56] you're saving your cost and time. So it [32:59] can or cannot have AI, but I think the [33:02] the focus should be how we can make our [33:04] system smarter uh with or without AI. AI [33:08] might be for today but maybe in in a [33:10] decade there might be a different [33:12] technology. So the focus should be on [33:14] intelligence. [33:16] >> I I want to say that one of the problems [33:19] we don't talk enough about when we talk [33:21] about moving fast is user acceptance. [33:24] Like this is what we do all day is like [33:28] make this new thing that we're very [33:29] excited about. Uh what other people are [33:32] doing all day is trying to use something [33:34] stable to get their work done. I thought [33:37] it was super interesting. I just saw uh [33:39] I think it was I forget who the article [33:42] on Microsoft has 500 million users who [33:47] have PCs that are capable of upgrading [33:49] to Windows 11 who are refusing. [33:53] And it's not like you can miss the fact [33:55] that they want you to upgrade. [33:57] And so that is very clearly a large [34:01] cohort of users saying I don't want [34:03] that. I what you have made is something [34:05] I don't want. And it's a kind of [34:08] telemetry that you know Rick was saying [34:11] we want to be able to read our customer [34:14] chats and and make sure we're not [34:16] drifting. [34:18] I just want to add I Googled Google [34:22] Windows as I'm a I'm an Apple user [34:25] because I don't like change of [34:26] experience. Uh Windows 11 came out [34:30] October 5th, 2021. [34:34] So that's not like it was new if you're [34:37] an Apple user and you didn't know when [34:38] Windows 11 came out if it's 500 million [34:41] customers. So that's some serious [34:44] reticence to change that we have to deal [34:48] with. [34:49] Um is it now also a security issue [34:52] because if there [34:54] >> they broke it they broke it. Um it [34:56] doesn't like a bunch of things don't [34:58] work right like core features don't work [35:00] right. Um, and they're like, "It's okay. [35:02] You can get around it with AI." And this [35:04] is just not a compelling case for people [35:07] who go and work in offices and do [35:10] spreadsheets. Like, [35:13] it I don't want it. I I just think that [35:17] that's a really interesting [35:19] it's not a guard rail, but it is [35:21] definitely a form of feedback and [35:23] telemetry that we could be watching is [35:24] like do people want to adopt as fast as [35:28] we can deliver or do we need to figure [35:30] out a way to deliver that's [35:33] more centered on the user experience? [35:37] And for Ricky, the user experience is do [35:40] you think it's your internal developers [35:42] as a platform engineer or is it your [35:44] external developers as you're moving [35:46] fast who has more reticence to change? [35:52] I guess a lot of the the big thing with [35:55] uh Wise's like key customer offerings [35:57] being apps and the website these are [36:01] very evergreen worlds now like apps have [36:06] been auto you have to forcefully go and [36:08] say you don't want auto updates for apps [36:10] now and websites have always kind of [36:13] been evergreen in the same fashion. So [36:17] like our general customer base is forced [36:22] upon that change. Now with engineers, [36:26] there's definitely a lot of stubborn [36:28] people. Um and it's very different. And [36:32] I guess what I [36:35] like is if you are actually selling [36:38] something really valuable, then you [36:41] should naturally see adoption and you [36:43] should then question if people aren't. [36:46] Is it that you've not hit the right [36:48] value proposition? Is it not as [36:50] impactful as you dream of? I think Heidi [36:53] said like we're building things that are [36:54] exciting to us. We're making them. So [36:57] sometimes you can be a bit more of a [36:58] deer in the headlights when it comes to [37:01] is it actually uh valuable for me. I [37:05] guess I'm always just looking for the [37:08] knock-on effects of if I can make Wise's [37:11] engineers more productive at the scale [37:13] we are now. That's a thousand engineers. [37:17] So if I can save them even 1 hour a [37:20] week, you're starting to talk about [37:21] massive numbers uh for wise. Um and like [37:26] AI has the power to do that. What I'm [37:29] kind of interested in is the most [37:32] looking at all like the industry [37:34] benchmarks that are coming out now is [37:36] the companies that would say they were [37:39] at least self-proclaimed elite with AI [37:42] in engineering. you know claim they're [37:44] getting things like 60% time savings. Um [37:48] these are numbers that like anthropic [37:49] claim for their engineers. Um so if that [37:53] is like the maximum if where wise [37:56] currently is is kind of a perceived [37:59] 2hour time saving. So how can I start [38:02] bridging that gap without change failure [38:05] rate going up which is one of our bigger [38:07] proxies for code quality. Um because I [38:11] can't have that number move. It's not [38:14] impossible to stop it moving. Humans [38:15] will always make mistakes even if we put [38:17] guard rails in place. But how can I keep [38:19] that number as stable as possible while [38:22] pushing PRs per engineer per week up? [38:26] It's a challenge. There's not all the [38:28] answers. We try and you can imagine [38:30] right now at wise 6 months ago barely [38:33] anyone had heard the word MCP server. [38:36] Then now that's out there. Then [38:37] anthropic almost claimed that MCPS is [38:39] not the answer. And so now we're looking [38:41] at our engineers using Claude skills and [38:45] it's moving so fast. You're just [38:49] constantly sprinting trying to keep up. [38:55] >> Absolutely. Is there anything else [38:58] that folks on the call should be looking [39:01] into that they may not be using yet, but [39:06] whether it's trade-offs around models or [39:09] maybe code freezes or maybe blue green [39:12] AB canary testing, [39:16] what do each of you think they should be [39:19] going back to [39:21] their teams or their organization and [39:24] prioritizing? [39:27] Dwan, you may go first. [39:29] >> Uh, I think like uh tying to what uh [39:32] Heidi mentioned that the what the user [39:34] wants, right? So for us, the the user [39:37] are people who build uh systems. So we [39:40] build systems for people who build [39:41] systems. So if a typical typical [39:43] developer onboards to a new project and [39:46] they have a giant codebase, they might [39:48] even know what question to ask. They [39:50] might get overwhelmed. But if you have [39:53] something like a backstage or internal [39:55] developer portal from harness, you can [39:57] onboard them at a much faster rate. Now [40:00] they can just hit uh create and then [40:02] they have the whole project and repo and [40:04] pipeline set up. Uh they can use AI [40:07] tools, their their AI code assistance to [40:09] make sense of the whole code base. So I [40:12] think that's what our user who are [40:14] developers want is make life as easy as [40:17] for me to make my first PR. I want to [40:19] understand the code base. I want to [40:21] raise a PR and I want to understand how [40:23] my code goes from my laptop to the [40:26] production because of course I can't [40:27] give my laptop to the code of course [40:29] what's on my machine [40:35] I see you nodding along [40:36] >> I guess that we are seeing like because [40:38] we try to constantly work out where we [40:41] getting the most impact from our AI like [40:45] um the because we're starting to spend [40:47] so much money now over multiple vendors [40:50] because people are scared to lock into a [40:51] particular vendor. Um, and I guess one [40:54] of the things we're seeing most valuable [40:57] is project on boarding especially in our [41:01] platform teams which are we are touching [41:03] so many more repos um from each other [41:07] and configuration everything like that. [41:10] We are seeing lots of PR uplift in [41:14] configuration type PRs and you know this [41:17] kind of work where engineers are able to [41:19] go into places they're less familiar [41:20] with change values and AI is giving them [41:23] a rapid qu like understanding of what is [41:27] going on and how it's working. So we're [41:29] seeing a big tick in productivity which [41:31] aligns with what Dwan says people want [41:34] is how do we get I guess the proxy [41:36] that's quite common is time to tenth PR [41:39] like how do we start shrinking this [41:41] numbers uh and I think it's a lot of [41:44] startups are always asking this how do [41:46] we get people to on board quicker you [41:48] know and this could be the golden ticket [41:52] >> fair enough Annie [41:55] >> thanks yeah I'll add to your point Rick [41:57] about you things are happening so much [41:59] more quickly. How do you keep up with [42:02] managing all that and making sure your [42:04] your change management is effective? So [42:06] something um from observability [42:08] perspective is I think this is a great [42:10] place to also use AI to help you in this [42:13] situation because if you're producing a [42:16] lot more changes there's a lot more PRs [42:17] going out it almost becomes a bit of a [42:20] fire hose to developers to kind of [42:22] manage all of that especially if they're [42:24] trying to say for example resolve an [42:26] incident or an outage and they don't [42:28] have as much context into what code is [42:31] actually in production because AI has [42:33] written it for example. So I think this [42:35] is it's a great opportunity to use an AI [42:38] agent to help you uh understand what's [42:40] going on in your system uh analyze all [42:43] your telemetry data and and offer you u [42:47] suggestions especially if it's connected [42:49] to your git as well can identify the [42:51] exact line of code uh and and things [42:53] like that. [42:55] >> Very good. hiding [42:57] >> documentation. What if [43:00] what if you could query your [43:02] documentation in a natural language way, [43:04] which is a very basic um use for AI, but [43:08] it pulls in all of the things you're [43:11] saying about like onboarding and speed [43:13] and being able to see what changes are [43:15] getting made. [43:16] The problem is that unsupervised AI [43:20] documentation is terrible and um it [43:24] affects a lot of people if it's wrong. [43:26] So supervised AI documentation I think [43:29] is probably the way to go and I think [43:32] it's probably going to be one of the [43:34] more powerful things that we pull out is [43:38] like [43:39] technical writers who can use tools in [43:43] order to deliver accurate information [43:46] rapidly [43:49] >> and it is something that the door report [43:52] overear not to be confused with the [43:54] European one they uh has shown that that [43:58] was the first thing that has made a huge [44:02] impact and [44:04] finding things is where developers waste [44:07] the most time and get frustrated and [44:09] break your physical hardware because [44:11] they've thrown it or something. They [44:13] can't find that thing. So, don't throw [44:15] things. There's no excuse for violence. [44:18] But it's the best way to close the gap [44:21] with onboarding like Ricky was talking [44:23] about time to onboard searchable and [44:26] it's also something that partnering with [44:28] AI would do well. It's similar with we [44:31] see that with firing customer service [44:35] and using chat bots failed mode using AI [44:40] not for the chat but mainly for the to [44:44] empower a super agent an AI and a [44:47] customer agent that can find information [44:49] faster that can scan like because a lot [44:52] of banks still have PDFs for their for [44:56] their uh purchase history and stuff that [44:59] So be able to scan what that person is [45:02] asking about and pinpoint [45:04] that increases your time to resolve and [45:07] your customer support. So look at them [45:09] as partners not replacement. [45:13] I am so grateful to each of you for [45:15] joining us today and to those listening [45:19] live or on the recording. I hope you [45:22] have a happy new year and we have lots [45:24] of good news coming about how we can [45:28] safely use AI to augment the developer [45:34] experience. So, thank you so much. [45:37] >> Thanks for