How Teams Test at AI Powered Velocity  ## Metadata - Author: [[LeadDev]] - Full Title: How Teams Test at AI Powered Velocity - Category: #articles - Document Note: --- tags: [Assign relevant tags or keywords to the notee to facilitate organization and retrieval. Add a # before each tag. Tags cannot have special characters. Do not use the tag #articles] - #AIPoweredTesting - #CloudNative - #QualityAssurance - #Testing - #CICD - #Observability - #ShiftLeft - #ShiftRight - #DevOps - #OpenSource - #WordPress - #OSPO - #ContributorExperience Author: Heidi Waterhouse (host); Panelists: Ole Lensmore (TestCube CTO), Nishi (Mendix), Vun Navasti (Software Engineer in Test/Automation) Source: YouTube / LeadDev webinar Reference: https://www.youtube.com/watch?v=RIqjQmmOeYQ date created: {{date:YYYY-MM-DD}} {{time:HH:mm}} --- # Title: How Teams Test at AI-Powered Velocity — Cloud-Native QA, Guardrails, and Collaboration ## Table of Contents: - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#-notes) - [📖 References](#-references) - [🗣️ Paraphrase](#-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB (Diversity, Equity, Inclusion, and Belonging) Assessment](#-deib-diversity-equity-inclusion-and-belonging-assessment) - [✅ Logical validity](#-logical-validity) - [🔍 DARVO Analysis](#-darvo-analysis) - [🔍 SEO Evaluation](#-seo-evaluation) - [🎯 Conclusion](#-conclusion) ### References: Include a list of sources consulted and cited in the article. - LeadDev Webinar: *How teams test at AI powered velocity* — https://www.youtube.com/watch?v=RIqjQmmOeYQ - Mentioned: DORA research report on AI & SDLC ([45:03](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=2703)) ## 📝 Summary: This LeadDev panel explores how **cloud-native teams** test at “AI-powered velocity.” Core themes: embed testing throughout the SDLC (shift left/right), build **guardrails** and **observability**, adopt **distributed/coach-like QA**, and use **AI incrementally** (requirement translation, scaffolding, reviews) while maintaining human oversight. Highlights include: integrated testing for microservices and ephemeral environments ([03:03–07:04](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=183)), testers as **coaches not gatekeepers** ([09:40–11:39](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=580)), strict CI/CD rules (no feature branches; no merge without tests) ([11:00–11:23](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=660)), AI-assisted onboarding/debugging ([21:15–23:10](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=1275)), and culture/process updates to prevent code bloat and context loss ([26:47–33:08](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=1607)). Management nudges like **unit-test coverage floors** and **buddy testing** cultivate a lasting testing mindset ([38:51–41:00](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=2331), [41:03–43:33](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=2463)). ## 📖 Main Content: - **What “cloud-native testing” entails.** Microservices, containers, infra-as-code, GitOps, and ephemeral environments demand **testing at every stage**; observability blurs into testing ([05:26–07:17](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=326), [12:00–15:00](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=720)). - **Shift left & right.** Bake tests early (contract/component/unit) and **measure quality in prod** with feature flags and chaos experiments where appropriate ([11:00–12:00](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=660), [12:55–15:01](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=775)). - **Testers as coaches.** QA partners with devs, co-owning quality; pipelines enforce rules (e.g., **no merge without tests**) ([09:40–11:39](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=580)). - **AI as an accelerant.** Use AI for scaffolding, translating requirements to test plans, code explanations, and skeptical PR reviews; avoid over-reliance and ensure team review/guardrails ([17:10–21:14](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=1030), [26:47–27:13](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=1607), [31:28–32:00](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=1888)). - **Culture & process.** Set coverage floors (e.g., 30%→higher), pair devs and testers, run **informal buddy testing**, and build a multi-year roadmap to evolve practices ([38:51–41:00](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=2331), [41:03–43:33](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=2463)). ### 🗒️ Notes - **🎤 Introductions & framing** — why testing now matters more than ever - Opening/housekeeping ([0:12–1:11](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=12)); goal: balance **speed + quality** ([3:03–3:09](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=183)). - Panel: Heidi (host), Ole (TestCube CTO), Nishi (Mendix PM/coach), Vun (SDET/automation) ([0:53–2:29](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=53)). - **🧱 Cloud-native = architecture + pipeline shifts** - Microservices/containers/Kubernetes; IaC/GitOps change **where/how** we test ([5:26–7:16](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=326)). - Testing needs to cover resilience, distributed network security, and e2e behavior ([6:55–7:16](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=415)). - **🔁 Shift-left & shift-right together** - No big “testing phase”; tests at all levels with **fast feedback** ([4:01–4:21](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=241)). - Testing in production + feature flags; monitor **quality metrics in prod** ([12:55–15:01](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=775)). - **🧪 Test types emphasis** - Contract/component tests built-in from the start ([4:49–5:07](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=289)). - Observability/telemetry as first-class test support ([14:18–15:01](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=858), [33:08–34:55](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=1988)). - **🧭 Roles & responsibilities** - Testers become **coaches**, not end-stage gatekeepers ([9:40–10:18](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=580)). - Hardening CI/CD rules: **no feature branches**, **no merge without tests** ([11:00–11:23](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=660)). - **🤖 AI in practice** - Incremental assistance: requirement→test plan, code explanation, plumbing generation ([17:10–20:04](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=1030)). - Onboarding acceleration and debugging support ([21:15–22:14](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=1275), [22:59–23:11](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=1379)). - Guardrails: team review; skepticism to avoid suboptimal “statistical average” code ([26:47–27:20](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=1607)). - Policy/process: naming conventions, coverage, contract-test counts, failure comms ([28:38–30:04](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=1718)). - Use AI as a **skeptical PR reviewer** that checks against documented guardrails ([31:28–32:00](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=1888)). - **📈 Scaling quality culture** - More AI-generated code ⇒ **more automated tests** to prevent pipeline clogging ([34:55–36:10](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=2095)). - Management strategy: start with **30% unit coverage** and ratchet up; 3–5 year evolution ([38:51–41:00](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=2331)). - Buddy testing, dev+QA **co-ownership** of features to internalize testing mindset ([41:03–43:33](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=2463)). - **🧠 Meta-insight** - AI is an **accelerant**: amplifies good teams and exposes dysfunction ([45:03–45:33](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=2703)). - Core principles of useful, repeatable testing don’t change ([47:04–47:17](https://www.youtube.com/watch?v=RIqjQmmOeYQ&t=2824)). ## 📖 References: - LeadDev Webinar: *How teams test at AI powered velocity* — https://www.youtube.com/watch?v=RIqjQ - Summary: [Music] Hello everyone and a huge thank you for joining us today for a discussion on how cloudnative teams test at AI powered velocity. - URL: https://www.youtube.com/watch?v=RIqjQmmOeYQ [Music] Hello everyone and a huge thank you for joining us today for a discussion on how cloudnative teams test at AI powered velocity. This is a lead dev webinar created in partnership with test cube. The webinar lasts roughly 45 minutes after which myself and the panelists will head over to the lead dev Slack to answer some of your questions in the webinar Q&A channel. We may also have time to answer a few questions. So, please submit those through the Q&A feature on Zoom and we'll get to them if we can towards the end. So, let's get started with some introductions. My name is Heidi Waterhouse and today I'll be joined by panelist experts in testing and QA. I'm really excited to do this because I think it's a thing that we don't talk enough about in our understanding of the software development life cycle. So they're going to share their experience and insights with you. I'll ask each of you to introduce yourselves starting with Ole. >> Thank you Heidi. Uh it's a pleasure to be here. Um, so I'm Ole Lensmore, the CTO at TestCube. Uh, and I'm looking forward to talking about the challenges related to testing and AI. >> Excellent. Nishi. >> Hey everyone. It's a pleasure to be here. I am a program manager at Mendix uh, based in Amsterdam currently. Prior to this, I've been an agile coach, a scrum master, a hands-on software tester, and also a corporate trainer on software testing and test automation related courses. So, I'm really excited to talk about all the challenges of testing today and Verun. >> Hey, hi, thanks for having me and uh myself Vun Navasti. I have more than 18 years of experience. My primary role had been uh started with the journey with manual testing then automation played various role from support uh now software engineer in automation um strategize and everything. So and I have couple of uh dozens like few not dozens few patterns on my uh on my plate. So yeah pretty uh happy to be here talking more on how AI can help and and automation how can we a leap ahead on it. >> Thank you. So this is a panel about testing and testing used to be a distinct stage of the software development life cycle and now it's a part of everyone's job. So how do we make sure that we have a unified approach to testing that moves at the speed of cloud native? We don't have a a separate part of our deployment process that says now stop everything for weeks to do testing. This panel has been assembled to discuss how to balance shipping quickly with maintaining quality. I think the first question I have is when we say cloudnative, what what do you think that means for our purposes for testing? Uh Nishi. >> Well, yeah. So to begin with cloud native earlier when we said cloud it meant we had some applications we then lifted and shifted them to cloud and say yes now we are now cloud right but now applications that are built and optimized for cloud environments are basically cloudnative so so they are being designed to be scalable in that way and resilient in that way and fast in that way and uh so all of this means we also need fast and automated feedback and that feedback comes in the form of inherent test uh tests at all levels and steps right so that's the reason we don't really have time space or capacity for a testing phase at the end >> what do you Fun. >> Um for now like the cloud native u is the collection of microservices or the application broken down uh into smaller chunks and we are deploying it fast. So for that we incorporate everything it uh on the cloud platform to deploy the application and for deploying application we should have inbuilt uh test which uh tests the application right from scratch. when you're building you're starting it you should think about it of adding uh your contract test your component test everything so that the speed is fast and which gives you the resilency and the speed which uh everyone so on on any failures or anything we are not there you get a feedback you turn around and the turnar around time is uh the devops model which usually follows in it which helps a lot so it's it's a quicker fast-paced and to have that cloud native helps you in uh to achieve that And Ole, you know, you're you're part of a company that does this. How do you think of cloud native and testing in cloud native? >> Um, uh, I mean, there's a lot of ways to look at this. So I think one is the purely the the architectural aspect of cloud native is that we're we're breaking down applications into microservices and containers and we're using container orchestration frameworks like Kubernetes and we might be running them in a hosted cloud or in our own infrastructure and and in parallel with that we're also adopting things like infrastructure as code and githops and moving to a different type of pipeline and I think um all of those things kind of re require us to think differently about testing and introduce new ways of testing and I think this is all also happening in parallel with the shift left which is something that's been going on for quite some time and then that also needs to kind of adapt to this and I think is actually a really good uh movement and maybe that's not a coincidence that uh shifting left is something that's really helpful when you break down your applications into smaller components and you try to test those earlier in the software development life cycle so I so to For me, it's it's about understanding the the architectural uh and infrastructure kind of um aspects of cloud native and then how does that then map to different types of testing? How do you inject those tests into your pipelines in in an efficient way and and how do you think about when you're leveraging cloud native infrastructure, you're maybe have like like Nishi said, there's an aspect of resilience. How do you test for that? And there's there's since your applications are often very distributed, how do you test the security of the networks that those things are distributed on? So there's there's another layer layer of testing that I think maybe wasn't as uh as relevant maybe you know before we did cloud native or 15 20 years ago. So I it's there's a lot in there when it comes to testing cloud native >> there. There is a lot in there. One of the things that I think about when we shift left, there's also the ability to shift right to enable and empower users more because we're giving them more control over what they accept as cloud customers. And I think that goes along with sort of the idea of observability and how it is that we are doing things that are not explicitly testing but are part of understanding what's going on in our systems. I think Verun, you said that one of the ways that we we need to do that is to break things down and understand them at a more granular level. Would would you say that's accurate? >> Yes. So uh to to detail it um so so as Olay said about u the infrastructure and the architecture um right now the problem is um as we know is the infrastructure is maintained by a separate team and when it comes to production or the lower environments it's always the importance is given to the production environment and not to the lower environment with the same capabilities what we have even though it's easy to scale up the application even though everything is in cloud so I guess uh that should give more power over there that like okay a team uh QA should be uh driven as a core with what development or we are trying to achieve it with infrastructure and architecture and everything but when it comes to uh QA the the unfortunate part is sometimes it's given less priority or not the exact production replica so I guess if they give that power of doing it I guess many times they would get the exact environment and even the HK scenarios anything which could break as a flaky flaky thing or which could be easily caught in the lower environments would help them give that uh power. So basically as just to add on it uh increase your coverage uh think from every every nooks and corners where you can include uh validation and have a same environment as that what you can have probably to to get a better product out of it and to use the optimal uh this native cloud over there. So, Nishi in an agile sort of understanding of the world, how do you think that moving to distributed testing changes the way we understand software? >> Well, it's it's not just about building software now. we we are baking in the quality that we intended at the end, right? So as testers become coaches, they are now helping uh that instead of just being the gatekeepers in the end, right? So it's more about what developers experience when they are expected to be testers every single day uh at every single step as well, right? How are they experiencing this change? How are they experiencing every single um uh step that is slightly different but slightly more complicated maybe but also slightly more important because what they do uh immediately rolls out and there's basically no time buffer no effort buffer that we have that used to be in the traditional context right so u if if like for example We sometimes had to harden our rules within our context. Like for our CI/CD pipelines that we were designing, we hardened the rules for ourselves. Like we would have no feature branches for any developer ever, right? So the the no feature branch rule. The second rule was the no merge without test rule, which which seems obvious to when we hear it, but in practice that wasn't ever the case. So we made life our ourselves harder but then with this we introduced a few extra steps in our pipeline that after maybe a few months felt natural and normal right so that kind of little changes in the experience made it easier uh in the end. >> Mhm. And Ole, did you have anything to add to that? >> Yeah. Well, so I wanted to go back what you talked about shift right because which is super interesting because we've talked to a lot of people uh you know in the space and one one aspect of shifting right is testing in production which is something that we see you you'd think like this is not something people do but people do it a lot and and it's something that they they shift left and they shift right and there's nothing left in the middle. Uh so they do uh because maybe because of some of the challenge they have with how do we automate tests in ephemeral environments sort of whatever that what's going on in the middle. So they have like the early tests to make sure that they're catching bugs early and then they have the safety net of testing in production and which is fascinating to me but it seems to be a pretty um wellestablished practice is maybe not is maybe the wrong word but it's something that at least we see a lot of people doing and then you measuring quality metrics in production and using that as feedback uh in back into the well early development life cycle. So it definitely and this is not something that they do exclusively, but it's also not something that they it's something they that's part of the QA uh um menu or whatever you want to say there. It's part of that uh adapting to cloud native. Maybe it is also because as you break down if you break down your applications into 100 microservices and those get shipped individually, maybe it's really hard to test an application in its real state before it actually goes into production. So there has to be some there's some gate at the end just to make sure and then you can you can debate what is monitoring and what is testing in production but I think that the what at least from what we see and some tests almost you have to do in production uh if like chaos testing is something that is something doesn't maybe make a lot of well I'm sure it makes sense but it has the best impact or the worst if you do it in production. So I think that's uh that's ju just wanted to to comment on that and then I wanted to comment what Nishi was saying interesting with not having feature branches uh uh and I think cloud native also maybe the cloud native technologies enable us to adopt those kind of workflows because uh thanks to cloud native and infrastructurees code and githops and those things it is actually possible to uh sometimes spin up environments for ephemeral use cases where you just want to test the that you're building. You don't need a full feature branch and all of that. So I think there's kind of a it's a there's a um pros and cons uh with everything of course but I think if le leveraging cloud native uh is not just about the end application but it's also about the infrastructure uh and the developer experience tester experience if I'm sure there's such a thing um uh and along your pipelines and and so I think there's just a lot there's a lot to gain with adopting cloud native uh uh across the entire life cycle. There is I Honeycomb has t-shirts that say test in production or live a lie and uh I feel like that's that's probably accurate but I also know that they are working with many people who have architected uh feature flags in so that they can you know shard off parts of production >> affect other people. It it's a really interesting way to think about this. So there's an elephant in the room that I haven't mentioned yet. I wanted us to sort of get our definitions under us first. And that's AI. And when Nishi talks about training developers to do QA and coaching them, I think a lot about how we're doing prompting and writing prompt templates. And I'm wondering if that's a thing that we need to be thinking about as you know part of our QA is how our prompt templates and our our guard rails work. Did have you worked with that Nishi? >> Well uh I was uh working with developers uh and coaching them much before AI came in. So at that time what we were working with were just our experience but in my experience good testers are good coaches by default. >> Even it's it's not about just communication skills or speaking skills. It's the inherent expertise uh of testing that makes you a good quality coach instead of just like a gatekeeper. So if if we were to bring in that alongside uh what and use it alongside the AI uh in the current realm that we are in what we uh recently did was used uh the expertise of testers to build some guard rails around uh design test cases, design business flows within a sort of a no code uh automation platform and use that can be used by developers uh product people business analyst whoever's other stakeholders on the team to all contribute in order to make good comprehensive and detailed automation frameworks. So we used the expertise but we then also leveraged other uh tools on top of it to make it into sort of an easy flow for everyone to say that we are doing automation on the go. So that that was one way but then uh we obviously need to also you know uh supplement it with some good uh coaching uh sort of around shared responsibilities also around risks risk areas risk analysis if we can do that within our um our context and then share it across as sort of like a playbook or you can say a testing handbook that everybody can U in the team >> Oley >> um well I I think it's fascinating to see how how people are starting are adopting AI uh across the entire software development life cycle but also in the STLC of course uh and the individual steps I think what I think what I find really important is is uh maybe to understand that for many it seems when people start using AI initially it's It's all it feels like it's an all or nothing kind of thing, right? Either I the AI generates everything or I'm not going to use it. But there's there's so many use cases in the middle of that where it can help you with the baby steps. It can, you know, it can kind of create the plumbing for you or it can break down things for you and it can explain things for you and it can compare things for you. I think that's and and and in in a in an extremely efficient way and especially in uh in testing you know it can translate maybe requirements to to a highle test case planning and it can tra it can from a test case description it can create an outline for a specific type of test and from there so I think um and then on top of that you'd want guardrails and maybe you want to uh prompt um templates like you were mentioning and there are even tools out there to help you capture like business rules in prompts and then apply those to your tests uh or the tests that you write or the uh the requirements that you create or the you know the test uh the test plans that you create. So as a kind of a as a governance framework and so I feel like um I think it's so interesting to see how the many different ways that AI can be used uh across the entire life cycle and obviously with test cube we're talking a lot about test execution but I think even earlier probably earlier in the STLC I think there's a lot of ways that yeah it can be leveraged and I I think people are still trying to figure that out and that's also one of the challenges everyone does it differently and there's no really clear kind of this is how you use AI because everyone uh is like I don't know this I prompted you know Claude to do this and it did it for me that was great and I'm happy and then they move on and so I don't know I um I don't know if that answers your question but I I would urge people to to approach AI with an open mind uh not go try to take two big bites initially and just see if how it can incrementally maybe help you in in the daily tasks that you have as a tester QA uh in your projects. >> So, Vun, how does how does AI change automation? Because Ole just said, you know, everybody does it their own way. Is that how you were experiencing it? >> Yes. So, for us, for me, I've been using uh AI very much day in day out now. Uh not in automation from your regular work uh in in our like for writing emails, writing feedbacks, uh responding them. And if you ask me with the coding perspective, yes, with coding, it has become much faster now. So like for example, anyone new is joining you, you explain them the high level, you pass on the code to them, you have an um like plugins out there. Uh like I use it, explain me uh this whole project, what is it about? Uh then creating test for example, I would just give okay, this is my uh interface and I need to create a contract respective to it. So it will suggest me something. Yeah, I need to be well understanding that it has u like it has speed me up what it used to take me couple of days now I can do in hours like creating component test it may be unit test I can verify that now the next level is so with the plugins with the new models what I use is I tell them okay I have written this uh code it's breaking on this point this is the error can you compile and let me know when I need to fix it or can you pin pinpoint me what exactly the root cause accordingly um I just run it sometimes it's a bigger code because there are limitations right now what every firm would be giving because it's it's the token based uh thing the pricing on the token probably so there is a limit right now I guess in the corporate world because it's pretty expensive but with that sometimes I just play it uh at night it does it job and then next I come back oh okay this was a solution okay now let me uh see it is a valid or So it's it's like now my things are very quick. I'm like I feel more equipped and I do like multitask like never before like even if you're managing and you have to be individual contributor it helps you out in every perspective in every angle which I'm sensing it out and u like appreciating it daily that okay this this okay it's just my power tool. It's my assistant I would say which is helping me without any um like no no no excuses anything like or not not excuses I would say without no limitation it it is like limitless the way I can use it I can use it it's it's my as said it's it's you it's free to use anything with what you can think of for the betterment of uh yourself I guess so for newcomers it helps for me daily it helps and everything and I'm huge fan of it um So yeah, pretty much >> that's great. I think that there's an interesting problem there, which is how do you synchronize everyone's AI use across the SDLC? Like do you find that people run into conflicts for Run? Like people have made things in different ways that aren't compatible with each other. >> U it's it's the way you design, right? sometimes uh it's it's not like the conflict I guess and when you work in a team um gratefully uh I have worked with teams where people are very open-minded so you you have to come with an open mind so the results would be there uh it the model is based on a particular data into it uh so so in the corporate health is if your model is in-house based which you are using it it would have a standard approach or the results would come as a standard approach But I have never seen in my team that if that approach is coming in if someone is having a better approach because they have been with that scenario or because they did the deep inside like their expertise is deep inside it people openly accept that okay this is the way better instead of this 10 line worth of logic I can do it in two line of logic let's do it this way okay if you're missing it so I don't see conflict conflict is like if conflict comes basically I would say people are not mature enough on that perspective to understand or not open enough or not a healthy environment over there. I would say if there is an um two-way conversation happening people would understand that okay this is because of this aspect and uh hopefully I have never came across that because if there is something uh like people listen uh they accept it or I accept it in way okay this is a better way let's go with that so just with an open mind uh go with u I guess the best approach or like I go or I have seen that till now in my DNA so yeah >> so Ole you looked like you had something to say >> yeah no I this is super interesting topic I think I mean something that we've seen is that um using AI to generate code you sometimes maybe are overly reliant on what it generates or or or you don't really know what you know it it seems to work. So this this is good and then I think what we've seen at least or I've seen is that somebody else then reviews that code and saying wait a minute what were you thinking here or that's this is maybe not you solved the problem but there are better ways to do this. uh and the AI is obviously trained on whatever is out there and it's going to use some you know statistical average of this is how you solve this problem which does of course doesn't necessarily have to be the best way to solve that problem in the specific context that you're in. So that's maybe a I don't know if it's a conflict but it's at least something that we've stumbled into. were like, "Oh, this is so easy and I just everything works now." And then somebody somebody else looks at it. Well, it and you you can of course let those things slip between the cracks. But in a year or two, you're going to get the risk is that you end up with a bloated code base where there are things in there. If it's just one occurrence, that's not a problem. But if you have a hundred or hundreds of small solutions to problems that aren't really optimal, I'm sure that will at some point also trickle, cause bigger issues for you. So I think um and so that's I don't know what the takeaway there is but I guess it is to still kind of be um uh maybe not critical but just be aware of that the what you're generating is might not always be the optimal thing and maybe discuss that with the team and have guardrails in place so people review and uh and and I mean obviously sometimes you have to let things go pass through because you're on a you know crunch or whatever. So, but it's it's just wanted to to comment on on things that I've seen as >> uh some uh kind of challenge. >> If you're enjoying this, then lead dev's got loads more to offer and there's tons to explore. >> So, the reason I love lead dev so much is they've got such a range of content on so many different topics in the world of engineering, leadership, and management. >> So, lead dev helped a lot on building my own career, right? And it's great to also see how it's helping the new generation as well. >> Make sure to sign up and join us at leadv.com. [Music] >> We're on YouTube. >> Yeah. No, I was at a talk yesterday where um the the people who have been doing people management and like high level management at this conference are super excited about being able to code again and pick up coding with Vibe coding and AI assistance. And uh this this guy was really excited that he had created in five days an app that had 60,000 lines of code. And I'm like, how are you gonna how are you going to maintain that? How are you going to like keep track of what it's doing? Like that's a lot of React, my friend. >> I I don't >> Yeah, >> like maybe that's just my misunderstanding of how powerful these assistants can be that they can also manage that. Uh but from a testing perspective, it makes me really kind of twitch. >> What do you think, Nishi? Yeah, I think it's all so new and so fast that none of us are equipped to actually answer how it's going to, you know, blow up uh after but but we see the signs already. But the but the thing here is that this is where some kind of guardrails and process comes in like putting some structure around individual elements that we expect from every uh individual on the team. Uh putting some structures around everything that we do which was not the case when we were just agile but but now it might be needed a little bit more. So uh structure around every single thing be it just uh how we check in, how we name things, how we uh structure things, how many lines of code, how many lines of comments to that level and then uh broader structures around uh how we actually get work done, how optimal it is, how uh unit tested it is, how many contract tests we have, how we integrate and how we guard when things fail. fail uh in one place or when contracts change in one place and how it is um communicated across to other areas that it affects right so we do need a lot more of uh that management around things u but yeah it's it's a process that we are all right now going through live right so it's going to be interesting to say the least >> does Does it feel like we're going to end up in a safe place to you, Nishi? >> Well, um, not unless we are as wary of it as Ole said, but then yes, we we do still need the enthusiasm and the acceptance that Warun expressed. Uh, it's just a balance somewhere in the middle. But uh yeah, every tool that we have ever adopted always had to have some limitations and some you know some guard rails around it and someone to say you know I'm a bit wary about it and that's a good thing and if if we have that kind of skepticism a little bit it helps. It really does. So let's let's keep our fingers crossed but but also you know not just hope but do things to make them better. I mean I think it's ahead it's just a fast it's just a super interesting use case for AI in itself like because you can instruct an AI to be skeptic skeptic and say hey for every code code that we pull request that we open I want you to look at this super skeptically you know you are a very skeptical senior engineer and point out to me how this is not in line with the guardrails that we've described in this uh document over here so you can definitely use a AI in those ways also. So, and I think that's really where where I find it fascinating that it's it's not just to generate, you know, your unit tests or your your playright test, but it's it's kind of augmenting the entire process of software delivery. Sorry, I interrupted you, but >> Yeah. >> Yeah. No, no, no, that's a really great point. I think it's it's interesting to think of it as as that mindset that you may not have. It's almost like having a a pair coder. Mhm. >> But yes, but one that's helping you >> keep in mind what needs to happen if you if you use it that way. Verun, uh when you're when you're using it for all these things, how do you double check that it's giving you what you want? >> Uh so basically foundation which helps. So everyone >> like I have a like uh from Q standpoint you have like the QA would have it's psychic thinking process okay what I should be validating what should be the approach what should be the edge case how it uh how things would impact and how he is going to validate but when it comes to the response back from this EI assistant probably um the foundation is we should be >> we should be knowing to read the code that okay if this is the expected output >> is giving me expected output. That's the first thing you would do. It's an optimized version, not optimized version. It's all later on it goes. But at the start, if you expect that it should be sum of two integers, it should be giving me that expected when I compile it compile the code. So the agent might compile the code. If not compile, you use that uh suggestion uh and try to compile it and see your outputs are there or not on first place. So go through it. Um and then yeah u it everything is uh it follows the step uh you run it you read it and if you're it works fine then you are there probably and then the second step would be how you optimize it what uh what's what's like how much is the output or what's the performance of it and all that's a later stage which you can measure with different testing uh matters when you try to deploy it on locally or or you deploy it on the lower environments and try to see the performance uh there's response time uh the the guardrails I would just put a point what we uh and Nishi spoke about it uh was uh for the 6,000 lines of code what we there if it's done the observability uh which if they do it in place that okay something is wrong uh the proper documentation which we can use by AI with the 6,000 lines we can properly document what what which section of it what it is doing we can document it out so when there are issues we can do that in code we can have logging in there where which would on observability in Splunk patterns on logging you would get an alert if something goes awire and you would exactly know where the issue would be in observability metrices you can see uh if there you are making a payment if it is broken where exactly it is broken you can see the component level things so if you have these guardrails in place and you know what you're doing I guess it it would be a blessing I would say >> yeah well it looked like you had something else to say >> no I was One of you mentioned uh seeing it as a as a pair of programmer or like a programming buddy. And I think that's a great way to think about it. What's also great about is is that these models are almost always in a good mood, right? So they're never complaining about they're always like, "Oh, good catch." Or, you know, "You're right. Yeah, I did something wrong." So there's never kind of this grumpy push back. No, nobody's angry or anything like that. You can always have a good good interaction with them hopefully. Uh so >> yeah, >> that's I think that's a good way of thinking about it. And then you can kind of yourself decide at what level you want them to to help. >> Yeah, that makes sense. >> So the other thing I was going to say is maybe downstream I think you were getting to this earlier on is that as we generate more code is this is something that we see there's also a need for more testing and as how do we both we're generating more tests. So how do we run those efficiently without clogging up our pipelines? uh but also we're generating more code and although we do our best to review that code like we discussed uh there's having real real having automated tests in place to make sure that especially in in in applications like uh vibecoded applications where context is often lost as the code base grows. So so the the the AI adds something over here and something over there breaks because it it's kind of lost the the understanding of that those things together. So there's definitely there a need for more testing to be in place. So that's also something that we kind of see as as the ripple effect of adopting more AI. And I think that's um yeah it' be interesting to hear if Ron and Nishi have similar experiences and how how how they would tackle that. >> Nishi. >> Yeah. So definitely the the loss of context um is one of the biggest problems that we will eventually face um if not already if we are just doing bits and bits with the help of AI and it eventually is going to lose some context somewhere and that's uh when your system's going to break and if if you are not um you know proficient enough in the system itself then you might lose the bigger uh game right so that's when uh having comprehensive tests and or some kind of checks placed at every single step that we do is going to help to find those in the right time otherwise yeah we are all going to be testing in production maybe so >> whether we meant to or not yeah >> yeah yeah so that that's the day we got avoid. >> So what helps people cultivate a testing mindset when they are developers? It's it's something that we've been trying to teach when we're like you you have to test that, you know, you have to do a unit test, you have to do an integration test before you can commit. But I think that's more like a compliance level and less like when I think of the the glory of testing, I think about, you know, people who are like, can I order negative one beer? >> Um, I think about people who are breakers in a really useful way. >> So, how do you think that in this world we can help developers cultivate that I wonder how I could break this mindset? Uh do you have anything for >> um that is a difficult task I would say uh to to cultivate that kind of like every everyone with the same mindset and vision that's why Ku plays an important role into it but I would say the guardrails of uh like from the hire management uh emphasizing that okay if you are if you want to deliver a code uh you need to have uh uh so So my so what I have seen the journey is for us it started with the slow pace of cultivating it from the management. So it's a strategy how they apply it. So it started okay uh you should have unit testing place okay 30 30% should be your uh mark you should be having it then there are sonar reports and everything which looks at your code base new line of code everything and this okay 30%. Once everyone reaches 30% the bar increases. So once the bar increases it automatically cultivates okay somehow may if I want to deliver the code I have to uh reach to that threshold. So once that threshold is there slowly people get into second stage okay now you have done unit testing now let's get back into contract testing contract there would be to many bureaucases because your he would be having 20 downstreams and 20 or more than like few hundred upstreams and downstreams. So your challenge would be to get everyone uh in in one place have their providers have their consumers everything getting into pack and then you're trying to do so it's it's it's not like bang on on day one you can do anything it's like uh you have to keep a goal of three to five year road map for it to cultivate uh or build or I would say like their DNA to get evolved. So you have to give that small small push in a smaller way so that it's it not come as a surprise to them because we are after all like I think as a human if something comes a surprise I'm like oh no I would I will I would push back a little bit even I I would tell them to do something right from day one 100% they would push back so it's not approach I would say slow and steady wins the race so uh uh road map of three to five year and then I guess things would turn around uh into it and people who are already there in the system they would know okay we have to do who are new are coming they would learn and do it probably so it's it's a cultivation >> yeah you had a you raised your hand Mishi >> yeah so I since uh Vun just talked about DNA um to evolution so what occurred to me the only two things that I've ever seen actually work in this regard of you know changing the developer's mindset and actually adopting testing in their DNA has been uh two things. One is pairing them up and actually buddying them with testers and then leaving them to it like they are co-owners of a certain part or a feature or a story or whatever way you do it and then leave them to it. So you work together. You do not uh you do log issues. You don't log issues. You test today. You test on the last day. We don't care. You figure it out. And when you do that and give that kind of ownership and they get their really their brains together, that's when they actually see each other's mindset. And that's when the true value of you know why I need testing right on the first day kicks in. And I've been at that stage. >> And then we had pods, agile pods which were essentially doing this but also adding a lot of you know uh selfsufficient but smaller uh level process within our agile teams. So what we did was buddy tests. Uh so without any formality around you know test reports defect logging or metrics of any kind we just did buddy tests for each other. So whatever you build, I just give you my opinion that this will break in x y and zed ways. And they are flabbergasted to tell you the least. Like uh and once that happens and starts happening again and again and again, it opens up the mindset of people of why really we need to include this as early as possible. Why really we need to shift leftest most possible. Right? So then it started changing the way we did user stories or did acceptance level tests at the time of thinking about stories and all of that right. So yeah I think uh that should be one option that we could take. >> Ole no I think those two were great examples. Uh so and they all based on on communication collaboration between people but because some people just don't have that mindset and some people just break stuff you know when as soon as they look at them and then they >> so have having someone like that on a team is amazing and and and getting them to collaborate with with with the people who have a harder time uh thinking outside that box is I think the probably the best way uh to try to ingrain that mindset. Uh, I'll just throw in a last uh another application for AI. It's a great way to say, "Hey, tell me some, you know, show me how you could break this code with some outlier use cases." And there's plenty of material that AI has been trained on when it comes to ordering minus one beers. So, it'll be reasonably good at at kind of coming up with stuff and even if those ideas are bad uh or not valid, they'll still get you thinking. So, uh I think that's also another kind of outlier use case where AI can be helpful. >> Excellent. Okay, I have one last question. We're we're gonna do this real fast. What book do you think people should read next? Uh, Ole, >> what book? Oh, wow. Uh, >> oh, it's a tough one. >> That's a tough one. Um, any book? I mean, is it like fiction? Oh, any book. Um, >> um, what comes to mind? Okay, I'm gonna cheat and go first. Yeah, >> the new Dora report just dropped yesterday and I'm halfway through it. It's not a book. It's only like 70 pages long. There's a lot of pictures. Uh, but it's really interesting because it's about how AI interacts with the whole software development life cycle and speed. And one of the key things they're saying is AI is an accelerant of whatever you're doing. If your team is broken, it will emphasize that it's broken. And if it's working well, it will emphasize that it's working well. And I think that's a really fascinating takeaway for us. >> Yeah, that that that is I don't have a recommendation, but uh didn't you recently write a book, Heidi? >> I did. I am releasing on November 4th, the progressive delivery book, which is about getting the right thing to the right people at the right time. And it's very exciting. We're doing a book signing tonight at ETLS. And it's been so fun to talk about how we can help people adapt their development teams to the fact that it turns out humans don't like change that much and even if we can deliver software very rapidly uh not everyone wants to accept it that rapidly. Yeah. >> Did you have any Thank you, Nishi. Uh, did you have any recommendations, Ruin? >> No, I I have been busy with kids. >> So, yes, >> I'm not reading it from at least uh 6 months, like at least a year now. >> So, I need to get back into the habit, but um not getting that time right now. >> Yeah. All right. Well, thank you all. We are all audience we are all going to be in the uh slack for lead dev in this in the room for this uh panel. Uh we really appreciate you taking the time to be with us today and we hope that this has been useful to you. I think that we talked a lot about how testing is changing but the core tenants of making sure that we're shipping something useful and repeatable are never going to change. So thank you all for the good insights in today's panel and we'll be in the slack. So thank you for joining and we hope to see you next time. Thank you to my panelists. >> Thank you for having me. >> Was great discussion. Thank you everyone., How Teams Test at AI Powered Velocity, LeadDev, 01k697v5ysb4vd78yyb1t9ybjc.