Are you ready for agentic testing? 🔗 **Source:** [Are you ready for agentic testing?](https://www.youtube.com/watch?v=ERW5Wdd6z_U) > [!note]+ Document Notes > tags: [agentic-testing, ai-testing, automation, test-quality] > author: LeadDev > source: YouTube > reference: https://www.youtube.com/watch?v=ERW5Wdd6z_U > date created: 2024-12-19 14:30 > > # 🏷️ Title: Are you ready for agentic testing? > <!-- deep-read:sections start --> ## 🧰 Action Inbox - [H1] Evaluate current test automation for agentic readiness — assess existing tooling and processes #testing - [H2] Define clear intent and requirements for agents — improve ticket and specification quality #requirements - [H3] Research agentic testing tools and vendors — explore Mabel and similar platforms #evaluation ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [YouTube Video](https://www.youtube.com/watch?v=ERW5Wdd6z_U) - [LeadDev Website](https://leaddev.com) ## 📝 Summary - Industry experts discuss the emergence of agentic testing - AI agents that can autonomously design, execute, and evaluate tests beyond traditional automation - Key benefits include reduced cognitive load for testers, better quality detection without explicit assertions, and ability to learn from experience - Major risks involve false confidence from incorrect results, loss of human testing skills, and over-reliance on probability-based LLM outputs - Success requires clear intent, rich context, proper guardrails, and understanding that agents complement rather than replace human testers - The technology demands a shift from scripted automation to goal-oriented testing with agents that can reason and adapt ## 📖 Main Content Agentic testing represents a paradigm shift from traditional test automation toward autonomous AI agents that can think, learn, and adapt their testing approaches. Unlike scripted automation that follows predetermined steps, agentic testing involves giving AI agents goals and letting them determine how to achieve those objectives through exploration, reasoning, and validation. The panelists define agentic testing as self-directed automated testing that goes beyond pass/fail binary checks. These agents can approximate human tester skills including visual observation, contextual reasoning, and experience-based decision making. They can design tests, investigate failures, create test data, and even fix broken tests autonomously. Key advantages include the ability to detect quality issues without explicit assertions - recognizing when something looks wrong even if not specifically programmed to check for it. Agents can also handle cognitive overhead tasks like log analysis and failure investigation, freeing human testers for more strategic work. However, significant risks exist around false confidence from LLM-generated results that appear plausible but may be incorrect. There's also concern about skill atrophy among human testers and over-dependence on autonomous systems without proper validation. ## 🗒️ Notes #### ⏱️ [00:00:00](https://www.youtube.com/watch?v=ERW5Wdd6z_U?t=0) — [H1] Introduction to Agentic Testing Discussion - Key idea: Engineers want to incorporate generative AI into testing but lack consensus on safe implementation - Evidence: "Not a lot of consensus on how to do it safely, how agentic or autonomous the tool should be" #### ⏱️ [00:02:45](https://www.youtube.com/watch?v=ERW5Wdd6z_U?t=165) — [H2] Panel Introductions and Expertise - Key idea: Panel includes leaders from Mabel (9 years in agentic testing), testing veterans, and engineering managers - Evidence: "Mabel, we've been on the journey for about nine years to make testing more agentic" #### ⏱️ [00:05:30](https://www.youtube.com/watch?v=ERW5Wdd6z_U?t=330) — [H1] Defining Agentic Testing - Skills Approximation - Key idea: Agentic testing aims to approximate manual tester skills including seeing, reasoning, and contextual action - Evidence: "Our goal in the space has been to approximate what the skills are of a manual tester" #### ⏱️ [00:07:15](https://www.youtube.com/watch?v=ERW5Wdd6z_U?t=435) — [H2] Autonomy vs Scripting in Testing - Key idea: Agentic testing shifts from scripted automation to self-directed goal-oriented testing - Evidence: "The main theme would be when automated testing stops being scripted and becomes more like self-directed" #### ⏱️ [00:09:45](https://www.youtube.com/watch?v=ERW5Wdd6z_U?t=585) — [H1] Problems Agents Can Solve Beyond Traditional Automation - Key idea: Agents can go beyond test execution to design tests, investigate failures, and create data - Evidence: "Great opportunity for us to now go beyond what we would class as test automation... designing tests, writing test ideas, producing dashboards, creating data" #### ⏱️ [00:12:30](https://www.youtube.com/watch?v=ERW5Wdd6z_U?t=750) — [H2] Quality Detection Without Explicit Assertions - Key idea: Agents can recognize problems without being explicitly programmed for every potential issue - Evidence: "We don't want this situation where all the tests are green and the house is burning down" #### ⏱️ [00:15:20](https://www.youtube.com/watch?v=ERW5Wdd6z_U?t=920) — [H1] Context and Experience Requirements for Effective Agents - Key idea: Effective agents need rich context, ecosystem access, and ability to learn from experience - Evidence: "A good agent doesn't act like it's their first day of work at your company every day" #### ⏱️ [00:18:45](https://www.youtube.com/watch?v=ERW5Wdd6z_U?t=1125) — [H2] Memory and Knowledge Ecosystem Integration - Key idea: Agents must store observations as memories and access organizational knowledge systems - Evidence: "The agent makes observations that get stored like memories that are accessible in the future" #### ⏱️ [00:22:10](https://www.youtube.com/watch?v=ERW5Wdd6z_U?t=1330) — [H1] Specialized Agent Teams vs Single Solutions - Key idea: Future involves multiple specialized agents with specific training rather than one general tool - Evidence: "It's not going to be just one tool... you need a plethora of agents, very specifically trained" #### ⏱️ [00:26:30](https://www.youtube.com/watch?v=ERW5Wdd6z_U?t=1590) — [H2] Intent-Based Testing and Workflow Triggers - Key idea: Agents need clear intent and structured workflows with human oversight and approval gates - Evidence: "The agents need to start with intent... whether it's requirements or test cases or some definition of good" #### ⏱️ [00:30:15](https://www.youtube.com/watch?v=ERW5Wdd6z_U?t=1815) — [H1] Risk of False Confidence and Incorrect Results - Key idea: Major risk is trusting incorrect but plausible-looking results from LLM-based agents - Evidence: "False confidence leading us to just lack... that would be more harmful than actually having a bunch of failing tests" #### ⏱️ [00:33:45](https://www.youtube.com/watch?v=ERW5Wdd6z_U?t=2025) — [H2] Learning from Past Automation Transitions - Key idea: Must avoid losing human testing insights like serendipitous discoveries during test execution - Evidence: "We lost all that serendipity... humans were doing hundreds of other tests at the same time" #### ⏱️ [00:37:20](https://www.youtube.com/watch?v=ERW5Wdd6z_U?t=2240) — [H1] LLMs Generate Plausible Not Necessarily Correct Output - Key idea: Remember that LLMs are designed to produce plausible-looking text, not necessarily correct results - Evidence: "Their goal is to produce text that looks plausible. It's not necessarily to produce something that is correct" #### ⏱️ [00:40:30](https://www.youtube.com/watch?v=ERW5Wdd6z_U?t=2430) — [H2] Impact on Testing Roles - Efficiency Era - Key idea: Agentic testing won't eliminate testing roles but will change them toward more strategic work - Evidence: "We're entering the efficiency era... doesn't remove the need for testing or skilled testers" #### ⏱️ [00:43:15](https://www.youtube.com/watch?v=ERW5Wdd6z_U?t=2595) — [H1] Importance of Clear Requirements and Specifications - Key idea: Success depends on clear intent and well-defined requirements - "garbage in, garbage out" - Evidence: "If you can't understand the ticket, an agent ain't going to understand the ticket either" ## 📚 References - [Mabel - Agentic Testing Platform](https://mabel.ai) - [LeadDev Platform](https://leaddev.com) - [Atlassian MCP Server](https://atlassian.com) ## 🗣️ Paraphrase - Agentic testing represents AI agents that can autonomously design, execute, and evaluate tests by approximating human tester skills like visual observation, contextual reasoning, and learning from experience - Success requires giving agents clear intent, rich organizational context, proper guardrails, and understanding they complement rather than replace human testers who focus on strategic work - Major risks include false confidence from plausible but incorrect LLM outputs, skill atrophy among human testers, and repeating past automation mistakes of losing valuable human insights ## 📰 Newsletter **The Future of Testing: Are You Ready for Agentic AI?** Testing is evolving beyond traditional automation toward truly intelligent agents. Industry experts recently discussed how AI agents can now approximate human tester skills - observing, reasoning, and learning from experience rather than just following scripts. **Key Benefits:** - Detect quality issues without explicit programming - Handle cognitive overhead like log analysis - Learn and improve from past testing experiences - Go beyond pass/fail to nuanced quality assessment **Critical Considerations:** Remember that LLMs generate plausible-looking results, not necessarily correct ones. Success requires clear requirements, rich context, and proper guardrails. These tools enhance rather than replace skilled testers. **Getting Started:** Focus on defining clear testing intent and improving specification quality. Evaluate specialized agent platforms like Mabel that integrate with your existing toolchain. Resources: [Learn WordPress Testing](https://learn.wordpress.org/), [WP-CLI Testing](https://wp-cli.org/), [WordPress Developer Docs](https://developer.wordpress.org/) ## ✅ Action Items #### DevRel - Developer advocacy: Present agentic testing concepts at conferences, create demos showing AI-assisted test creation - Education: Develop workshops on prompt engineering for test automation, office hours for agent implementation - Developer experience: Improve testing documentation to support agent integration, update SDK examples - Developer marketing: Create content series on AI testing evolution, case studies from early adopters - Community management: Moderate discussions on AI testing ethics, gather feedback on automation concerns #### OSPO - Policy & Governance: Establish guidelines for AI agent usage in testing workflows, review contribution policies - License Compliance: Scan AI-generated test code for licensing issues, train teams on AI output compliance - Security Management: Assess vulnerabilities in agentic testing tools, create security attestations for AI usage - Software Inventory: Include AI testing agents in SBOM strategy, track ownership of generated test assets - Developer Support: Provide training on safe AI testing practices, recommend approved tooling options - Community Engagement: Fund upstream testing framework improvements, establish agreements with AI vendors - Strategic Alignment: Set KPIs for AI testing adoption, align OKRs with quality improvement metrics #### Community - Maintainers: Support testing roadmaps that incorporate AI capabilities, provide governance templates for AI usage - Contributors: Create onboarding programs for AI-assisted testing contributions, label first-issue programs appropriately - Sponsors: Maintain transparent budgets for AI tooling costs, report impact of agentic testing investments - Ecosystem: Coordinate cross-project standards for AI testing integration, share lessons learned #### WordPress - Core: Prioritize AI testing infrastructure, triage issues related to automated quality checks, set performance budgets - Core Performance: Use AI agents for profiling test coverage, establish metrics for testing efficiency improvements - Core AI: Implement guardrails for AI-generated tests, create evaluation sets for testing agent quality - Core Program Team: Coordinate agentic testing rollout across teams, manage dependencies between testing initiatives - Mobile: Integrate AI testing into app development workflows, improve telemetry for automated test results - Playground: Create labs environment for testing AI agents, support reproducible bug reporting with agents - Community: Develop meetup kits on AI testing practices, support WordCamp sessions on automation evolution - Training: Create lesson plans for AI-assisted testing, develop learning paths for test automation skills - Marketing: Campaign messaging around WordPress testing innovations, develop narratives on quality improvements - Support: Establish SLAs for AI-assisted issue resolution, update knowledge base with agent-generated insights - Design: Conduct UX audits using AI agents, implement design tokens that support automated visual testing - Documentation: Fill gaps in AI testing guidance on DevHub, create style guides for agent-generated content - Accessibility: Use AI agents for automated WCAG compliance checking, implement tooling for accessibility audits - Meta: Integrate agentic testing into .org platform roadmap, ensure API reliability for testing agents - CLI: Expand command coverage for AI testing workflows, improve documentation for automated test scaffolding - Hosting: Develop cross-host testing checklists using agents, implement telemetry for testing effectiveness #### The WPCC - Programs: Fund fellowships focused on AI testing research, provide microgrants for testing tool development - Projects: Establish governance clinics for AI usage policies, support documentation sprints for testing guides - Funding: Create matched grants for testing infrastructure improvements, set milestone payouts for quality metrics #### FAIR - Technical independence: Ensure metadata capture for AI-generated tests, implement provenance checking for automated assets - Policies: Maintain transparency in AI testing tool funding, establish anti-capture policies for vendor relationships - Governance artifacts: Create charters for AI testing usage, develop operator agreements for agentic tools ## 🧪 Hypotheses - Agentic testing tools will be most effective when given clear intent and rich organizational context rather than operating autonomously - Teams that maintain human oversight and validation of agent-generated tests will achieve better outcomes than those relying on full automation - Organizations with well-defined requirements and specifications will see greater success with agentic testing adoption - The effectiveness of testing agents will correlate with their ability to access and learn from historical testing data and organizational knowledge ### 📊 Methodology - Comparative study measuring test quality between human-supervised agents vs fully autonomous agents across different application types - Longitudinal analysis of testing effectiveness before and after agentic tool adoption, measuring both defect detection and false positive rates - Survey research on tester skill development and job satisfaction in teams using agentic testing tools - Case study analysis of successful agentic testing implementations across various organizational contexts ### ❓ Questions - How can organizations effectively measure and validate the quality of agent-generated tests? - What specific skills should human testers focus on developing as agentic tools handle more routine testing tasks? - How might agentic testing change the economics and staffing models of QA teams? ### 🔮 Future Research - Investigation into optimal human-agent collaboration patterns for different types of testing scenarios - Development of standardized metrics for evaluating agentic testing tool effectiveness and reliability - Exploration of agent specialization strategies and multi-agent testing orchestration approaches ### 💡 Implications - Practice: Testers need to shift from writing test scripts to defining clear testing intent and validating agent outputs - Policy: Organizations must establish governance frameworks <!-- deep-read:sections end --> ## 📺 Video  --- # 📄 Full Transcript *Auto-generated from YouTube captions · timestamps roughly every 30 seconds · [watch on YouTube](https://www.youtube.com/watch?v=ERW5Wdd6z_U)* **[00:07]** Hello everybody and uh thank you for joining us today. We'll be having a discussion asking are you ready for agentic testing? Um so engineers everywhere are trying to incorporate more generative AI into their testing workflows and there's been a broad push to do this but not a lot of consensus on how to do it safely. um how agentic or autonomous the tool should be um or even exactly what agentic testing even means. So we are here to get into some of those **[00:37]** details. This webinar will last roughly 45 minutes. Uh and after that both myself and the panelists will head over to the lead Slack to answer any remaining questions in the panel Q&A channel. Uh and we might also have time to answer a couple of questions live. So, please submit them throughout the webinar using the Q&A feature on Zoom and we'll get to them if we can towards the end. So, without further ado, um let's get started with some introductions. Uh my name is Amanda. I am an engineering **[01:08]** manager at Data Dog and I'll be moderating today. Um and I will pass things over to Dan to do an introduction. >> Thank you, Amanda. Uh great to see you all. I'm Dan Belchure. I'm one of the founders at Mabel. Uh at Mabel, I lead our product engineering and uh technology organizations. Uh just a little bit about uh us and we can get into more detail later. Um Mabel, we've been on the journey for about nine years to make uh testing more agentic. And so this is a a topic that I have a a lot of **[01:41]** passion about. I'm really happy to be uh joined by some uh amazing colleagues on this on this topic. Amazing. Thank you. And Nishi, you want to go next? >> Sure. Yeah. Hello everyone. I'm Nishi. I've been into the software industry for about 18 years. I've been a hands-on software tester, a trainer on test automation and low code uh test automation programs. I've been also into **[02:12]** learning and enablement for various product companies. And currently I work as a program manager at Mendix to also help coach um the agile teams as well as lead the learning and community programs there. So yeah, I'm super glad to be here. >> Glad to have you. Richard, can you do a quick intro next? >> Sure. Hey everyone. Um so I'm Richard. I've been in testing QE for about 19 **[02:42]** years I believe now. um currently a senior architect in quality engineering at Slalen. Very long title, but for the last year I've been building and testing um agent AI solutions. Um so that's yeah, that's what I've been up to. >> Great Vernon. >> Hey, greetings. Uh I'm Vernon. I'm a senior engineering manager at a company called Phys. And we're a a language uh technology company. We build a platform. **[03:13]** So you can localize all of your content very easily and at scale. Uh and when I'm not doing that, I um speak at conferences and I do a little bit of consulting and I have a podcast with my pal Richard who you just heard from. So yeah, that's how I keep myself out of trouble. >> Awesome. Thank you. Uh great. So to open things up, we'll start with a roundroin question. Um so I'm going to pose to the group. What does agentic testing mean to **[03:44]** you? >> We should pick Don. Go on, Dan. >> Oh. >> Oh, I can start. Yeah. Yeah. I mean, I think I think our goal in the space um has been to uh approximate, you know, what um the skills uh are of uh a manual tester. Um, and so, uh, one of the things that we focus a lot on is making sure that that tester can not only do things but also see things and reason with context and experience on **[04:15]** applications. And so, to me, those are some important attributes of an agentic tester, the ability to see quality, reason, um, you know, take action and, uh, yeah, I'll stop there. >> Richard, do you want to go next to pick on you? >> I pick on me. Um I I yeah I could read the panel two ways. The first one that comes to mind is you know are we ready for testing agentic systems or even agentbased systems for a start and then the second one is kind of the tools that **[04:47]** people like Mabel are building. Are we ready for them and kind of how they fit into the existing approaches that we're taking to testing because they're very different. So I think we definitely have a place for agents in the way that we test. I think we just need to understand better what they're good at and what they're not so good at and how that blends with existing approaches people take for quality and testing. >> Definitely. Vernon, what's your opinion, **[05:17]** >> man? Um, I would I would say agentic testing is is a little is about how autonomous is the automated the automated testing that you're doing. So you've got you've got your agents and you can you know build specific ones to do specific things but then if you get like an army of those things or a group of those things and you just give them a mission to go and complete they will just go and do that somewhat independently of you. >> So that's that's that's my understanding of what um agentic testing and and you **[05:48]** can imagine all the kinds of trouble and shenanigans that you could get up to when we start to uh use that kind of thing. Yeah. >> Yeah. Nishi, what do you think? >> Yeah, exactly. So, to me too, the main theme of agentic testing would be when the automated testing that we've all been talking about for so many years now stops being scripted and becomes more like selfdirected. Instead of us writing scripts, we are **[06:19]** just giving it a goal and it is now running it completely with reasoning, with exploration, with validation, with reporting mechanisms, relearning stuff on top of that and then you know the whole loop back again. So it's more about the autonomy but alongside the still the automation of of things, >> right? Yeah. And there's some question **[06:50]** of how much structure you provide and to what extent um an agent is kind of acting within bounds or acting fully autonomously, right? >> Yeah. >> Cool. Um yeah. Yeah. And then I guess more broadly, uh, what problems do you think that an agent could solve for you? Um, so without getting too much into maybe specific setups, what are some of the qualities that we think an effective agent should have? >> I don't know about qualities, but when **[07:21]** you said the types of problems it can solve, I think it's a great opportunity for us to now go beyond what we would class as test automation. that was very much I want you to execute this exact steps and tell me a result at the end. Because we have the advantage of LLMs, we can now have agents designing tests, writing test ideas, producing dashboards, creating data for us. There's so many permutations that have been possible but not to the extent they are now. So when I when I think of the possibilities, I think we need to start branching beyond test automation. But **[07:53]** even within that space, there is options now to fix tests. But the agents that I've been building are more away from that and more like investigating broken builds for me. Why do why has this test failed? Go and research all the plethora of things that I don't have time to do and just give me an opinion. You know, it's not fact, but give me an opinion on what you think might have gone wrong. So, I think there's all those little nuances now that we can bring all that information into an agentic agent and have it start guiding us a little bit. Not not fully, but you know, nudging me **[08:24]** in a direction. It it's it's mainly about taking some of the cognitive things, the cognitive load of a tester like Richard just said, like the the things that I don't have time to do, do it quickly and then let me make the decisions on top of that. So maybe investigations that I want to do uh but I don't have 45 minutes to just run through all these logs and then find out the one thing and uh things like that **[08:56]** basically make uh testers uh mental load lesser in some way. >> Yeah. The one thing I'd add there is just a to me a really fascinating place where agents can you know really exceed what we see with traditional automation is um you know not requiring assertions for everything right so like you know we don't want this situation where all the tests are green and the house is burning down right the agents should be able to recognize when there are you know **[09:26]** problems without just like testers without being explicitly told what a problem you know every problem entails. Right. So there's not only a much broader class of problems, but we can potentially even rely on agents to find those problems themselves. >> Richard, I saw you unmuted there. Did you want to add something? >> I think it's trying to tie together what Dan and Nishi said. It's that it's that cognitive load, right? Like agents like we always hear they can think, right? **[09:57]** They can't think, but right, they can make sense of stuff for us, especially if we give them that guidance. We've not been able to do that with traditional test automation. It's either right or wrong, pass or fail. >> Whereas now we have the ability to get, you know, opinions or um conclusions around certain bits of data that we can give it. And that, like I said, that opens the door to so much more potential tooling that we can build that hasn't been possible before. >> Definitely. Yeah. And to go back to I guess that second part of the question. **[10:28]** Um what kind of qualities would an effective agent have >> and maybe Vernon if you want to chime in on this one. >> Man, I'm not I'm actually not sure. Um at the risk of anthropomorphizing the tool, I I I like I almost think of like what what does what does a human tester need to be good at? Yeah. >> And it needs to be the same. >> Yeah. >> The thing is **[10:59]** >> it can't >> it might get a bit gnarly there because it it like it isn't a human. Like it it doesn't it doesn't know what it feels like to be pinged when production is down in the middle of the night. But because it's been trained on a ton of data, it might be able to make suggestions or, you know, interpretations of of what may or may not be happening and what's the cause and that kind of thing. So, um, the quality the qualities I would want it to have is I'd want it to have >> I'd want it to have the not as much un **[11:32]** understanding. It's really difficult to talk about this thing like it's like it's not a human. I'd like I'd like to have an understanding of the context that we operating in >> you know our customers and who we are building it. And then it will I think it would give much better >> results or output if we can if we can give it that information. >> Yeah, totally. And I think yeah, that's kind of the crux of the problem is recognizing that this thing is not a human and trying to find the things that it's good at um within that. And this is **[12:05]** a good segue actually. Um, so yeah, my next question is how do we give agents enough context and guard rails and guidance to actually make them effective? >> I know Rich has got opinions about this. >> Go ahead. [laughter] Go on. >> Um, I think those are two those are two two separate things and those two things are massive in themselves, right? So context like you know prompt engineering got renamed context engineering for a very valid reason right there's so much **[12:39]** nuances to the data that we have and that it can reach that we we need to give it that when we talk about framing the context. So, a lot of us wh why why is it that if someone tells me I don't know um someone told me that Dave wrote the code I'm immediately like oh I've got to test this a bit more than usual right whereas if someone was like Jane's written this code I'm like oh it's Jane it's going to be fine right there's those little nuances of that context or certain tickets that have a thousand comments on all those rules need to kind **[13:11]** of be codified if if they exist in your place and that's the next problem how do you know they exist. How good at you at understanding your own context to actually explain that to an agent to help it understand? We're not very good at realizing why we're good at things. Right? So when you're trying to train an agent to be good at something, it becomes quite difficult. And then once you've done that, then you're like, actually, you're really good now, but you might be a bit too good. So I need to I need to keep you on this straight and narrow now, and you shouldn't be going over here. So it's it's a real **[13:43]** challenge. A bit like Vernon said, if you could imagine a human tester in your team, like a very very junior one, what would they need to know and how would you keep them going in a certain path? And they are they are two huge challenges. >> Yeah, I I think context is is king in this. Um, one thing specifically that I think is important is that the agents can't a good agent doesn't act like it's their first day of work at your company every day, >> right? Which is kind of what you get **[14:14]** from an LLM, right? Just raw LLM. They need to be able to um gain experience, right? Which is basically building context over time, right? So, I understand this application. and I understand I have existing automation that I can use right rather than you know learning it like it's new every single time right so I would separate out the kind of skills like what does it mean to be a good tester with you know as I um show up tomorrow that I've **[14:45]** learned something you know uh yesterday >> and Dan does that mean some amount of proactivity also is that kind of what you're describing >> um well kind of specifically what what I'm describing is um a a world the case where the agent makes kind of observations that get stored like memories that are accessible in the future, right? This is where the vector embeddings and and so forth come from. Um and so that to me that's really what **[15:17]** you know distinguishes a an in you know really intelligent agent from one that you know uh is challenged to approximate a tester is whether you're kind of really experiencing what it's like to test and using that you know making that information available um as you continue to test as the agent right >> yeah I guess >> go ahead please >> oh yeah and some of that comes down to the tool you're using right and the capabilities Okay. And maybe it'd be useful to talk a little bit about Mabel **[15:47]** here actually. >> Yeah. I I think it's the um Mabel this is core, right? Is if if we're building a tester, the tester has to be able to, you know, learn just as you know, we've all learned in our in our testing. Um but the importantly, you know, that learning in Mabel becomes accessible um to the agent in just the same way that your memories as a tester, you know, become accessible to you. so you can get better. Uh and then the other thing is the ecosystem of knowledge, right? So if **[16:18]** you're a tester, you have access to Jira and you, you know, you use that as a core part of your workflow and you have access to GitHub and you have access to, you know, all this other context. And at Mabel, we think it's it's been really a breakthrough to make sure that the agents have access to that context, right? And so when we think about data and context as part of our, you know, testing efforts, it's really important, I think, to choose a tool that can tap into the ecosystem of knowledge that your testers, you know, rely on to be **[16:49]** effective. So that's effectively what, you know, what we've done and continue to do is, you know, build agents that can um help us accelerate testing kind of at scale by tapping into, you know, the knowledge of the people on the team and then the other systems that you have. Thank you. Yeah. Yeah. And Vernon, I see you nodding. Anything you want to add there either on I guess the context side or also guidance and guard rails. >> Uh yeah, that Dan got me got my brain **[17:21]** going off in all kinds of mad directions cuz when he when he said he talked about the agent getting experience, so Rich said, "Oh, it's it's like a this junior tester." And then Dan made me think, well, how long will it stay junior? And you know, do we need to articulate that as well? And then do we need to try and see this is going to get weird. So, do do we need to try and instill something that looks like humility **[17:52]** or is that is that what the guard rails are for? To kind of stop it to contain it from just going >> completely, you know, terminate on on everybody? I don't know. >> But I liked I like I really like the whole experience thing that I hadn't thought of that before. That's a really got me thinking there. That's pretty cool. >> Nishi, what do you think? Yeah, I think u what Dan said there like agents to be really smart need the firstly yes access to all the other **[18:23]** systems the entire ecosystem of knowledge that a tester has but and add on top of it a little bit of context of also the past test history let's say all the logs ever of everything and also u let's Jira histories of uh similar issues that have happened in the past and who were they by and who resolved them and how uh it happened right and all of that which is taken very granted as Richard said like **[18:55]** we don't know how we are we are such a good tester or an experienced tester would actually not know how to teach the same context to a junior tester let's say so the same thing applies here but until You can actually say well this AI uh is actually helping me write tests but how long is it only going to help you write tests? It should be agent can help me test and that is it. And once it **[19:28]** gets to a tool for writing tests from this to actually helping you test in the truest form that's when it's actually smart enough uh you know so until then you obviously need those guardrails you obviously need some boundaries around it otherwise it can go all over the place. >> Yeah definitely. So yeah, I mean on the one hand we're talking about giving a tool a lot of autonomy to you know not **[19:59]** just write tests but also determine what the problems are and kind of ultimately lessen our cognitive load. >> Um but then on the other hand we also want this tool to have something maybe something like humility to kind of know the bounds of its knowledge and operate within that. >> Yeah. But I think that I think that nails kind of what Dan was saying about what Mabel's doing is I think going forward is it's not going to be just one tool, right? Test automation, a lot of companies try to solve it with one or two tools. But when it comes to agents, **[20:29]** I think you need a plethora of agents, very specifically trained, know what bits of the context to pull out, have very specific guard rails about and then very specific instructions about how to do things well >> in your context. And I I would love to know from Dan like it must be a challenge of like Mabel needs to know stuff right but that stuff is different in every company right it's different tooling it's different structures it's different data structures it's got to be a challenge to unify that to get the **[21:00]** value from such an agent >> yeah absolutely just like a tester showing up you know you change jobs you show up to a new job and there's new new expectations um I think the the one key thing is you the the agents need to start at least in the enterprise where we tend to focus with intent right whether it's requirements or test cases or you know some definition of good right they have some native understanding of that right **[21:30]** like a 404 page is probably bad everywhere right but >> but you know really when they're learning how to test the application >> the the team needs to provide some intent. That could be requirements, that could be user stories, that could be test cases because just like a tester, sometimes you want to tell them that success looks like, you know, these completing these very granular steps. Sometimes with a more experienced tester, you want to say, you know, I just just make sure I didn't break the **[22:01]** checkout flow, right? And uh and so I think the agents need to be able to adapt to, you know, different levels of intent, but it has to start with some intent. Yeah, part of that is building I guess like the rich ecosystem of context, right? But then it sounds like part of it is also giving specific guidance like Richard was talking about kind of building a team of testers and they all have maybe a different purpose and a specific set of instructions. >> Yeah, absolutely. Just like when you're using, you know, an LLM and you say you **[22:33]** are a blank, right? I think that, you know, it's helpful to tell the agents that you're an accessibility tester or you're particularly sensitive to, you know, uh, design related issues and and what have you. >> Provide like a frame of reference [clears throat] >> or skills. >> Um, Nishi, did you want to jump in there? I saw you. I I just also wanted I'm curious about it now that Dan explained it also a bit like how do you also look at it from a workflow point of **[23:04]** view like do you give it uh autonomy but then also triggers that okay with a workflow like let's say every time somebody uh there's a PR request you go ahead and do this this and this and then uh find out something and then report back and only then we then approve or reject further down the line. Right. So is there a workflow somewhere? Is there a trigger somewhere? And is there human involved somewhere in this entire thing? **[23:36]** >> Yeah. Yeah. The the human of course starts with the intent, right? We we still rely on the you know people to express what the the goals are certainly. Um I I think one of the interesting things in this time um it's getting easier and easier to for example automate things just like in previous generations say it got easier to uh provision computers right when virtualization and cloud came in and what we see is when things get easy we end up doing them too much and it becomes very expensive right and so I **[24:07]** think you know the the the question of you know how do you um trigger these things is really important right because you don't want to you spend millions of dollars having thousands of agents working 24 hours a day, you know. Um, and so there is a life cycle component where, for example, with Mabel, um, you would you would describe specific things that you want validated with every deployment and then others that you want validated, for example, as part of a broader regression cycle and the agents follow those instructions. Right? That's **[24:38]** the orchestration >> sort of side of it. The the thing that I think is fascinating is we're seeing more and more, you know, the definition of done sometimes is agent to agent, right? You're using, you know, claude uh to do some, you know, development workflow and you need a tester to be, you know, paired with that developer and, you know, together they'll decide when it's done, which I think is a really um exciting and and you know, timely uh topic. >> Yeah. and goes back to kind of that **[25:09]** specialization that Rich was talking about where each agent has kind of a unique purpose and maybe its own intention. >> I hope you're enjoying this video, but did you know lead dev has a lot more to offer? >> When I think about lead dev, I think about bricks and hearts. Bricks are those foundational things that you need to know and lead dev provides that through webinars, through videos, through cutting edge research. But hearts are the things that you need to glue yourself to the people at work and lead does that through conferences and **[25:41]** meetups. [music] >> Make sure you check out leadv.com. >> You got it. >> Cool. Um yeah, and then I guess kind of opposite of this. So we're talking a little bit about context and guardrails and human guidance. um what are kind of the risks in this space and particularly what risks do we not appreciate enough in this context and I'll start with maybe Burden to pick on you Jen **[26:15]** >> um I think I don't know if it's a risk or not but I think I think the com I think the conversations that we have about you know whatever aspect of AI In testing, it always seems to gravitate towards what we currently understand automation testing to be, which is some binary checks that resolve down to a yes, no, pass, fail, true, false. And I think that I think there might be if you if **[26:46]** you if you think more broadly about it and think about automation in testing and where we can use tooling to do the other stuff that are useful in testing like creating data for example. Is there is there are there any other aspects of the work where we could use use these tools? Um so that's uh so that's one thing. I think another thing, and this is something that Rich and I have talked about um on our podcast a little bit, is **[27:18]** um is is task is task analysis. So like how how do we how do we know that the things that the agents and the a agentic AI that they're correct? >> Yeah. >> But how do we know that they're right? How do we know that they're effective? How do we even evaluate that? How do we even ask that question? you know, in the same way that we'd say, "Hey, human, are you doing a good job with your testing?" What does that look like when it's a when it's a robot, basically? Um, I don't I don't know what the answer is. **[27:48]** Um, but yeah, they're they're the things that spring to mind for me. >> Yeah, Richard Nishi, do you have thoughts on that side? Kind of like evaluating. Yeah. Yeah. That's that's what my worry too like the incorrectness of what we get from the typical charg responses like will that also translate into our agentic tests and how can we validate that that how can we trust that then and then that's **[28:20]** more harmful than actually having a bunch of failing tests because that would be okay false confidence of okay let's push to production and then let it all bombard out of control. Right? Um that's I think the top most on my worry list of risks. Um but yeah, another one would also be the uh the false confidence **[28:50]** uh leading us to just lack now now that we have agentic testing. What about testers losing um context, losing skills, losing interest in in that bits that Agentic has taken up and now we we are basically losing it, losing control over it. >> Yeah. And I think this is one of the most difficult parts about working with an LLM in a lot of contexts, right, is kind of knowing when to step in and when to kind of back off. And it's easy to **[29:21]** mistakenly put too much trust in whatever query or tool that you're using, right? And then get burned by it. >> Yeah. Yes. >> Rich, I see you nodding a bit. >> Yeah. Yeah. The there's so I can think of loads of risks, but like the biggest one that comes to me is I don't want us to forget the learnings from the past. So we had the transition going from manual test cases to automation. And in that transition, most people realized that the human was just doing a a fraction of what the test case said. So **[29:51]** they did what the test case said, but they also did hundreds of other tests at the same time. And that they they never spoke about that and that never got recorded, right? It was pass fail. But now when we when we started automating all those tests, we lost all that serendipity. >> And now I think with agents, we have an opportunity to bring some of that back. So we have the ability to say yes the basic test does pass the very clear intent but also what else did you see while you happen to be on those pages just like the human used to >> and now I think if we can do what Dan's **[30:22]** suggesting and we can oh nearly said close the loop that's such a terrible thing to say but if we can like if we can get some of this like results verified by a human and have the agent improve over time which isn't how I'm seeing a lot of agents be built but if we can we can start to like like start to trust some of the output a little bit more, but I think in the short term, exactly what Vernon said, we we can see the agent was satisfied the intent, but how did it do it? What steps did it take? Did it do the right steps? Did it **[30:54]** skip a whole half a page of the UI and just do it via the API because it learned there was an API? And it's things like that that that we talk about the guard rails. It's making sure we're framing these toolings, keeping them in into a certain approach and training them well enough. They might not really train them, but putting the right prompts in and guidance. >> So, I think if we don't do that, we run the risk of people believing that this is auto magic again, right? These agents are all powerful and they've just tested my whole website. And it's like, well, they didn't. They only thought there was two pages and it tested two pages, **[31:26]** right? [laughter] That's that's not we don't we can't have that. So I think we just need to be sensible and realize that testing is not easy and we prove that with automation and now we need to continue that knowledge into the agentic space and keep it people's expectations level. >> Perfectly said. >> Vernon, did you want to add to that? >> Uh a little bit. So we should remember that all of these Gen **[31:59]** AI tools or at least the textbased one. So if you're talking about writing code and things like that, they are all LLMs under the hood. And that means their goal is to produce text that looks plausible. It's not necessarily to produce something that is correct. And so we need to remember that and not you know not dify the output too much or just forget to question it. So I think I think that's the big risk is that it's it's its job is to you know give us **[32:31]** plausible looking output not necessarily correct output. If we remember that we stand a better chance of being successful I think. >> Right. And that goes back to, I guess, Rich's point that we need to kind of understand the limitations, but also maybe ideally give enough guidance that an agent could have something like intuition to be able to understand like, okay, I'm testing this button, but really the page kind of looks off in general, right? >> Yeah. That's See, that's the thing, **[33:02]** isn't it? It's like it [clears throat] it won't it won't actually have intuition. And I said humility earlier and we've all used human emotions and words to describe this thing, but but what it is is is it's a probability machine. So it might look like intuition, but it isn't intuition. But that's not to say it's not useful. So it's not to say, oh, this is just all smoke and mirrors. Let's not use it. It's just it's more it's more around, you know, what is the tool good at? What is it bad at? Where can we leverage those characteristics and attributes most in the work that we do? And I think **[33:33]** we'll be golden. Yeah, Dan, what do you think? How does this kind of relate to your experiences? >> Yeah, I I completely agree. I think it goes both ways, though. So on one hand I think this you know era of automation that Richard you know highlighted um you know took us down a path where sometimes in the automation we say let's say I'm searching for a blue couch that the definition of correct is that you know there's some element with inner text **[34:04]** that says blue couch and [laughter] and and you know if that is true even if there's the picture of the blue couch shows like a missing image then the test passes, right? And the nice thing about agents in the modern era, you could say, >> "Please just make sure that the results are reasonable and that there's a blue couch and it knows that this is a blue couch that showed up and it looks for a correct label, right?" So, um, I think, you know, we're making a significant step forward with Aentic compared to **[34:34]** where we were, you know, in the manual or phase one of automation. Um, but I think to Vernon's point, you know, we still have a ways to go and there's a ton of value in, you know, testers being more strategic and, you know, um, we don't need you to check for the blue couch, right? We can handle that with the agents, but there's much more nuance there. >> Yeah. >> Can I just add to that? I think Dan's made a really good point and it's the like the types of assertions we can now do have not been possible before but **[35:05]** also how many people are thinking about those types of assertions right so you might be building them into tools like Mabel but like people now need to change their thinking of like you said what does good look like and historically we've not been very good at saying what good looks like it's like I've passed this test and as I said with the test cases the test case says can you buy a sofa but as a human I made sure the image was there. I made sure the price was correct all the way through. I did all those things, but I when I write my automated tests, >> well, I I do this, but not everyone **[35:36]** does, right? When they write their automated tests, it's a race to the end. And then you get to the end and it's do what? Does it says order confirmed? So, again, that's my concern about that thinking. So, the options are so much stronger than they've ever been, but are we equipped to take advantage of it? And will will we is the question. But Dan's right, the oracles and assertions that we can think of, we need to get that testing mindset going again and train and guide these tools to be good at that. **[36:07]** >> Yeah. And that brings me, I guess, to my next question. Um, so what are kind of the implications of this new era on testing roles and responsibilities in our industry? Broadly, >> they're all they're all gone. No, I'm joking. [laughter] >> That's that's that's the quote. That's the quote we want, right? But they're all gone. Um, as with every other tool and it just changes, right? So again, people are going to have to adapt, right? This technology is here and we should take advantage of it, but we should do it in that, you know, structured and learn take a learn **[36:38]** approach to it. Some people still haven't, you know, have embraced automation. So like, you know, it's another tool that's out there. It might not work in your context, but there is an there is an element of a risk if like again I'm not saying you don't want to say things like this, right? But if you are doing really basic testing that you know the type of stuff we said a minute ago that you're just checking there's some text on the screen and you're not good at communicating that you're actually doing more than that. There is a risk, right? But that risk was the same for automation. And if you **[37:10]** navigated that risk, you're going to probably navigate this one too, right? But I think it's about what we just said about changing your thinking, your approach, and realizing that there's a new plethora of tools at your disposal now. And don't be scared of them. Embrace them, play with them, try them out, see what works, see what doesn't work. I don't see it replacing people's jobs, but it might free you up to do more. It might mean that you don't need 10 testers anymore. You need seven. Right? So, the work won't go away, but it's going to change it. And me and my **[37:41]** Vernon keeps mentioning the podcast and Vernon said something really cool on the podcast which was we we're seem to be entering the efficiency era when it comes to AI and I think we should think about that in testing. How can we make ourselves more efficient with the use of automation and agents and agentic approaches and that might lead to a change in the way teams are structured but doesn't remove the need for testing or skilled testers or critical thinkers. It just might mean that we we we take **[38:11]** some of what we're good at and give it over here and we we learn new stuff to do. Um so yeah. >> Yeah. Vernon, anything you want to say there? >> Uh man, tons man. So I think I think I think econ I mean I think there's oh man I think there's there are economic decisions being made about on the on the on the basis of what people think this technology can do **[38:42]** which mean people are losing their jobs that may be happening to a larger degree or a lesser degree I don't know but those conversations are are definitely happening and I think you know depending on the tool vendor a a vendor of vendors might be positioning this in in very directly saying if you if you use our tool you won't need these these pesky humans over here so we can get rid of them >> that that that does I've had people have told me this like vendors have said this to me directly so I think I think **[39:15]** because those conversations happening there is a risk to job security but I think if you if you get a A way to mitigate for that is to is to kind of do what Rich said and and I'm sure everyone else on the panel will advocate for. It's like really understand what how to be effective in the role and it's more than just >> binary yes no pass fail true false. **[39:46]** There's there's way more to it than that. So, if you can impact that and start to really double down on those skills, I think as Rich said, you'll be able to, you know, navigate your your way through. Um, yeah, I think I think I think and another thing that I that I'm seeing people talk about more recently, one of those people is uh Kelty High Totower, who is an absolute legend in my eyes. Um he's the first person I heard talking about well **[40:17]** I've se people talk about the best the people who are getting the most out of these tools are are are senior folks. If you compare the effectiveness and output of people with experience and people who are more more junior engineers testers whatever these people who've been more long in the tooth in the career are getting better outcomes with the tool. So the question becomes if it is true that companies are hiring fewer juniors **[40:48]** it means there are less people that are going to be getting that experience. So in a few years time where are the seniors going to be coming from like that's an issue we need to be thinking about that. How do they get that experience? How do they how do they do that that junior level grunt work in order to become senior in order to get the most out of the tools? No idea, but it's something we should be thinking about. >> I'll stop. **[41:20]** >> Thank you, M. >> Um, yeah, we have just five minutes, so I want to jump to a couple of the questions that people have asked. Um, so I think the first one that came in was can we get some examples of specific setup? So I kind of said at the beginning, let's not get too much into specifics. Um, I think yeah, Dan, you had talked a little bit about kind of Mabel. Um, any other anything else you want to highlight there? Any other tools that we should call out? Oh. Um, as far as specific setups, um, **[41:51]** well, I guess I'd say a a canonical setup with Mabel. Um, is that you have uh uh test case uh or test cases in Jira. Um, you have access to some uh assistant large language model like claude or copilot. um you uh integrate um Jira with Copilot using uh uh the Atlassian MCP server and you integrate Mabel with Copilot using **[42:23]** the Mabel MCP server and then you say things like uh create an automated test for uh test case Jira 29 and then Mabel will go and automate the test case and then you could use that as your interface to communicate with Mabel's agents from a you know execute some tests or what have So I guess that's a a a common setup on the agentic side. I hope that is helpful. >> Yeah. And then actually quick piggyback on that, someone else had a question about Jira tickets specifically. Um so level of related documentation can vary **[42:55]** significantly in terms of quality and level of detail. Um would you or would anyone else recommend kind of a standardized approach to writing jer tickets or some other kind of analogous tool um in order to help agents be more effective? I don't have a standard approach, but the thing that Dan said at the start, the intent needs to be clear. So like the intent of what that ticket's for, what they're trying to build and the definition of done or what good looks like. >> They they kind of need to be there **[43:26]** otherwise it's going to guess. So I don't have a standard layout for you, but like something to go and research is the Oh, it's got a name. Dan might know the spec. there's like spec driven stuffs coming back and people are suggesting girkin again which really makes me sad but like there's got to be a better format than girkin but something like that with very clear specs like I think is where we're probably going to end up when it comes to writing tickets that would allow coding agents to better understand it would allow testing agents to better understand but that spec also needs to **[43:57]** remain human readable so I think right now there isn't one but I think there's definitely smart people looking at that >> um but right now I would say and like keep tweaking tickets. If you can't understand the ticket, an agent ain't going to understand the ticket either. >> Yeah, I think it it works both ways because if we are talking so much about how to set up guard rails for our agents, then the same guardrails would then apply to back to us first because it's human-driven after all even if it's **[44:27]** agentic, right? So if we uh set a guardrail for it to understand X Y and Zed, we first have to also put that guardrail on everybody across the teams to define things in X Y and Zed formats for the agents to understand. So yeah, it it starts in the very beginning as we keep saying it's shift left and agentic is like the leftest possible shift that could ever be. So yeah, it it starts in the in the specifications and goes all through to the end. **[44:59]** >> It's garbage in, garbage out. >> Yes, absolutely. >> Okay. Yeah. And I actually think that's a great note for us to wrap on. I know we didn't get to all questions, so we [music] will jump into the Slack after this. Uh but I want to thank all of our wonderful panelists for their input and thanks to all of you for watching and actively contributing questions. Um really hope that you enjoyed this session and hearing from this panel. Um so please head over to the panel Q&A channel next and we can continue the discussion there and thanks for joining. **[45:31]** We hope to see you next time. >> Bye everyone. Thanks everybody. Bye everybody. See you.