How Tekion enabled 1,300 employees to work with agents 🔗 **Source:** [How Tekion enabled 1,300 employees to work with agents](https://www.youtube.com/watch?v=RjFUUIeoybI) > [!note]+ Document Notes > tags: [ai-agents, developer-productivity, enterprise-ai] > author: Lee Dev > source: YouTube > reference: https://www.youtube.com/watch?v=RjFUUIeoybI > date created: 2024-12-19 10:30 > > # 🏷️ Title: How Tekion enabled 1,300 employees to work with agents > <!-- deep-read:sections start --> ## 🧰 Action Inbox - [H1] Implement persona-driven AI framework — structured approach with human-in-loop validation #ai-strategy - [H2] Deploy code review AI in GitLab pipeline — immediate L1 review with checklist scoring #code-review - [H3] Create telemetry dashboard for AI adoption — track quality scores and productivity gains #metrics ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [Tekion](https://tekion.com) - [Augment Code](https://augmentcode.com) ## 📝 Summary - Tekion deployed a persona-driven AI framework across 1,300 engineers with 50-85% productivity gains - Framework includes specialized AI agents for requirements, architecture, development, testing, and code review - Built custom telemetry dashboards tracking quality scores (94%+) and human bandwidth savings - Started with May 2024 evaluation, scaled company-wide by June with CEO/CTO support - Uses structured checklists and human-in-loop validation to maintain 80%+ AI accuracy while embracing the remaining 20% for creativity ## 📖 Main Content Jawahar, Senior Director of Engineering at Tekion, led an AI transformation that moved beyond typical 30-40% productivity gains from raw LLM usage to 80-100% gains through a structured persona-driven framework. Tekion operates three cloud platforms serving automotive dealerships, OEMs, and third-party partners. The breakthrough came when Jawahar realized that large tasks needed architectural breakdown and that AI agents performed better with specific, bounded responsibilities. Rather than fighting model hallucinations, the framework embraces 80% accuracy targets with human oversight for the remaining 20%. Key innovations include automated code review AI integrated into GitLab pipelines, comprehensive telemetry tracking quality scores and adoption metrics, and planned "headless orchestration" where AI teams operate with minimal human intervention. The system maintains context through Augment's engine while ensuring governance through mandatory checklists for each persona. ## 🗒️ Notes #### ⏱️ [00:00](https://www.youtube.com/watch?v=RjFUUIeoybI?t=0) — [H1] Introduction to Tekion's AI transformation - Key idea: 1,300 engineers now use AI agents in daily workflows through persona-driven framework - Evidence: "Started as small internal experiment, today more than 1300 engineers use AI in their daily workflows" #### ⏱️ [03:45](https://www.youtube.com/watch?v=RjFUUIeoybI?t=225) — [H2] Tekion's three-platform architecture - Key idea: Automotive Retail Cloud (DMS), Enterprise Cloud (consumer-dealer-OEM), Partner Cloud (APIs) - Evidence: "Automotive retail cloud is primarily the DMS... brings end consumers, car dealer, and OEMs under one platform" #### ⏱️ [05:23](https://www.youtube.com/watch?v=RjFUUIeoybI?t=323) — [H1] Timeline of AI adoption at Tekion - Key idea: Q2 2024 evaluation to company-wide rollout by June 2024 - Evidence: "Q2 2024 started evaluating tools... Come June 2024 we rolled out augment code across the company" #### ⏱️ [07:15](https://www.youtube.com/watch?v=RjFUUIeoybI?t=435) — [H1] Discovery of persona-driven framework necessity - Key idea: Raw LLM usage led to hallucinations and prompt fatigue, requiring structured approach - Evidence: "My entire day was about prompting... hundreds of prompts. That's not how we can do on day-to-day basis" #### ⏱️ [09:30](https://www.youtube.com/watch?v=RjFUUIeoybI?t=570) — [H2] 80% accuracy target with human-in-loop - Key idea: Framework targets 80% AI accuracy, leaving 20% for human judgment and LLM creativity - Evidence: "Any AI framework should never target 100% accuracy... targeting 80% plus through strong prompts" #### ⏱️ [12:45](https://www.YouTube.com/watch?v=RjFUUIeoybI?t=765) — [H1] Complete persona workflow demonstration - Key idea: End-to-end flow from Jira ticket through requirements, architecture, development to code review - Evidence: "Starting from Jira ticket... requirements AI, architect AI, planner AI, developer AI, code reviewer AI" #### ⏱️ [16:20](https://www.youtube.com/watch?v=RjFUUIeoybI?t=980) — [H2] Checklist AI validation system - Key idea: Each persona has mandatory and optional checklist items with scoring thresholds - Evidence: "Checklist AI comes into picture... scores based on mandatory and optional sections" #### ⏱️ [19:45](https://www.youtube.com/watch?v=RjFUUIeoybI?t=1185) — [H1] Adoption strategy with aggressive goals - Key idea: Started with 50% AI implementation target, scaled to 100% with leadership support - Evidence: "Next sprint we going to do 50% of tickets using AI... next sprint 100% AI only implementation" #### ⏱️ [23:10](https://www.youtube.com/watch?v=RjFUUIeoybI?t=1390) — [H1] Code review AI integrated in GitLab pipeline - Key idea: Automated L1 code review with immediate feedback and inline comments - Evidence: "Every MR that's being raised gets automatically L1 reviewed by our code reviewer AI" #### ⏱️ [27:30](https://www.youtube.com/watch?v=RjFUUIeoybI?t=1650) — [H2] Telemetry dashboard showing productivity gains - Key idea: May to September comparison shows 46% reduction in human comments, increased throughput - Evidence: "May: 810 merged MRs, September: 1477 MRs... 46% reduction in human comments" #### ⏱️ [31:15](https://www.youtube.com/watch?v=RjFUUIeoybI?t=1875) — [H2] Quality scoring and auto-learning system - Key idea: 94.69% average quality score with plans for AI auto-learning from comment patterns - Evidence: "Code reviewer AI average quality score is 94.69%... implementing AI autolearning based on acceptance rates" #### ⏱️ [35:45](https://www.youtube.com/watch?v=RjFUUIeoybI?t=2145) — [H1] Future roadmap: headless orchestration - Key idea: AI teams operating autonomously with manager AI and minimal human intervention - Evidence: "Headless orchestration... scrum team of AIs only... manager agent automatically picks features" #### ⏱️ [38:20](https://www.youtube.com/watch?v=RjFUUIeoybI?t=2300) — [H2] Recommendations for enterprise AI adoption - Key idea: Define structure first, integrate into workflows, treat AI as team members with accountability - Evidence: "Starting with structure is very important... making AI first class team member with clear metrics" #### ⏱️ [40:15](https://www.youtube.com/watch?v=RjFUUIeoybI?t=2415) — [H3] LLM as judge validation approach - Key idea: Using LLMs to judge other LLM outputs through challenger patterns and self-critique - Evidence: "Very much effective... challenger pattern, self-critique pattern... using two different LLMs helps" ## 📚 References - [Tekion Company](https://tekion.com) - [Augment Code Platform](https://augmentcode.com) - [LeadDev Resources](https://leaddev.com) ## 🗣️ Paraphrase - Tekion created specialized AI agents for different roles (requirements, architecture, development, testing) rather than trying to make one AI do everything - They embraced AI limitations by targeting 80% accuracy with human oversight, rather than fighting hallucinations through endless prompting - Success came from treating AI agents like team members with defined responsibilities, checklists, and accountability metrics integrated into existing workflows ## 📰 Newsletter Tekion's persona-driven AI framework shows how enterprises can scale AI beyond individual developer productivity to transform entire engineering workflows. Their approach of specialized agents with structured checklists achieved 50-85% productivity gains across 1,300 engineers. Key takeaway: Rather than fighting AI limitations, embrace them with 80% accuracy targets and human-in-loop validation. The real breakthrough comes from treating AI agents as team members with specific roles and accountability. This structured approach could transform WordPress development workflows, from automated code review in Core contributions to specialized agents for accessibility testing, performance optimization, and documentation generation. Resources: [WordPress Core Handbook](https://make.wordpress.org/core/handbook/), [Code Review Guidelines](https://make.wordpress.org/core/handbook/testing/), [Performance Best Practices](https://make.wordpress.org/core/handbook/best-practices/performance/) ## ✅ Action Items #### DevRel - Developer advocacy: Create case studies demonstrating persona-driven AI adoption patterns, present at WordCamps on enterprise AI transformation - Education: Develop workshops on structured AI frameworks, office hours on checklist-driven development - Developer experience: Integrate AI code review into Core contribution workflows, improve documentation with AI assistance - Developer marketing: Campaign highlighting WordPress AI tooling ecosystem, content strategy for enterprise adoption - Community management: Survey community AI usage patterns, moderate discussions on AI ethics in development #### OSPO - Policy & Governance: Establish AI contribution policies, review intellectual property implications of AI-generated code - License Compliance: Scan AI training data sources, ensure compliance with WordPress GPL requirements - Security Management: Review AI-generated code for vulnerabilities, establish attestation processes for AI contributions - Software Inventory: Track AI tools in development stack, maintain SBOM for AI dependencies - Developer Support: Training on AI governance, consultation on enterprise AI adoption - Community Engagement: Fund research on AI impact in open source, establish agreements with AI platform providers - Strategic Alignment: KPIs for AI adoption quality, OKRs for productivity measurement #### Community - Maintainers: Governance templates for AI-assisted contributions, roadmap integration with AI tooling - Contributors: Onboarding programs for AI-assisted development, first-issue programs with AI guidance - Sponsors: Transparent budgets for AI infrastructure, impact reports on AI productivity gains - Ecosystem: Cross-project coordination on AI standards, shared AI agent development #### WordPress - Core: AI-assisted issue triage, performance budget validation through automated agents - Core Performance: AI profiling for optimization recommendations, automated performance regression detection - Core AI: Evaluation sets for Core AI features, guardrails for AI-generated content - Core Program Team: AI coordination workflows, dependency analysis through AI agents - Community: AI-powered meetup content generation, WordCamp speaker matching algorithms - Documentation: AI gap analysis for DevHub/HelpHub, automated style guide enforcement - Accessibility: AI-powered WCAG compliance checking, automated accessibility test generation #### The WPCC - Programs: AI fellowship programs, microgrants for AI accessibility tools - Projects: AI governance clinic development, documentation sprint automation - Funding: Matched grants for AI accessibility research, milestone tracking through AI agents #### FAIR - Technical independence: AI-powered metadata validation, automated provenance checking - Policies: Transparency requirements for AI usage, anti-capture mechanisms in AI governance - Governance artifacts: AI-assisted charter generation, operator agreement templates ## 🧪 Hypotheses - Persona-driven AI frameworks achieve higher productivity gains than general-purpose AI tools in enterprise environments - 80% accuracy targets with human oversight produce better outcomes than pursuing 100% AI accuracy - Checklist-based validation systems can maintain code quality while scaling AI adoption across large engineering teams - Integration into existing workflows (GitLab, Jira) drives higher AI adoption rates than standalone AI tools ### 📊 Methodology - Controlled study comparing persona-driven vs general AI tools across similar development teams - Analysis of quality metrics before/after checklist implementation in AI workflows - Longitudinal tracking of adoption rates with integrated vs standalone AI implementations ### ❓ Questions - How do persona-driven AI frameworks perform across different programming languages and technology stacks? - What are the optimal team sizes and organizational structures for implementing enterprise AI agent systems? - How can open source communities adapt enterprise AI governance models while maintaining contributor accessibility? ### 🔮 Future Research - Development of standardized enterprise AI adoption maturity models - Investigation of AI agent coordination patterns in distributed development teams - Exploration of AI governance frameworks for open source software communities ### 💡 Implications - Practice: Shift from individual AI tool adoption to systematic AI workflow integration - Policy: Need for AI governance frameworks balancing automation with human oversight - Security: Requirements for AI-generated code review processes and vulnerability management - Education: Training programs must cover AI agent coordination and validation techniques - Funding: Budget allocation for AI infrastructure and specialized agent development ### 🧠 Learning Styles Assessment Reading level estimate: Flesch-Kincaid Grade Level 12 #### 👀 Visual learners - Present: Workflow diagrams, dashboard screenshots, telemetry visualizations - Improve: Architecture diagrams showing AI agent interactions, visual persona relationship maps #### 🔊 Aural learners - Present: Video format with speaker narration and live demonstration - Improve: 5-minute audio summary highlighting key framework components #### 🗣️ Verbal learners - Present: Clear definitions of persona types, framework terminology, detailed explanations - Improve: Glossary of AI agent terms, analogy-driven explanations of complex concepts #### 🧩 Kinesthetic learners - Present: Live demonstration of GitLab integration, hands-on dashboard exploration - Improve: Step-by-step implementation checklist, interactive framework builder #### 🧠 Logical learners - Present: Structured workflow diagrams, metrics tables, quality score calculations - Improve: Decision trees for persona selection, causal diagrams showing productivity impacts #### 🧑‍🤝‍🧑 Social learners - Present: Team adoption stories, collaborative framework development discussions - Improve: Peer review processes for AI implementations, community adoption guides #### 🧘‍♀️ Solitary learners - Present: Individual framework assessment opportunities, self-guided implementation paths - Improve: Personal AI adoption worksheets, reflection prompts on framework applicability #### 🌿 Naturalistic learners - Present: Ecosystem thinking around AI agent coordination, organizational environment considerations - Improve: AI lifecycle management principles, sustainable adoption strategies ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading structure, descriptive dashboard element labels - Improve: Enhanced alt text for workflow diagrams, ARIA labels for interactive telemetry displays #### Keyboard navigation - Present: Video player controls accessible via keyboard - Improve: Ensure dashboard demos include keyboard navigation patterns #### Contrast and readability - Present: High contrast in presentation slides and dashboard displays - Improve: Verify 4.5:1 contrast ratio in all telemetry visualizations #### Text resizing and reflow - Present: Content maintains structure at increased zoom levels - Improve: Test dashboard responsiveness at 200% zoom #### Media and captions - Present: Video content available with detailed transcript - Improve: Time-synchronized captions, downloadable transcript format #### Forms and interactions - Present: Clear error handling in dashboard examples - Improve: Accessible form <!-- deep-read:sections end --> ## 📺 Video  --- # 📄 Full Transcript *Auto-generated from YouTube captions · timestamps roughly every 30 seconds · [watch on YouTube](https://www.youtube.com/watch?v=RjFUUIeoybI)* **[00:06]** Hi everyone and welcome to Augmented Engineering in action, the series where we show you how companies are really using AI. My name is Scott Kerry. I'm the editor-in chief here at Lee Dev and I'm going to be your host for this fifth and final session of this series in association with augment code. I'm joined again for this session by Matt Bull from Augment. Matt, how you doing today? >> Good, thanks Scott. Thanks for asking. Um excited for another one of these these sessions particularly with Jawaha someone that I've worked with a little **[00:37]** bit already and uh both Jawahar and and Techon really thought leaders I think so excited to get into this one and learn more. >> Yeah, I've been really impressed with some of the stuff Jawahar has been showing us behind the scenes. So I'm really excited to show everyone else what we've been learning about over the last couple of weeks. Um so yeah without further ado let's introduce our our uh panelists today. So Javahar um hi welcome. Could you just tell us a little bit about yourself and the work that you do at Tachon? >> Sure. Thanks Scott. Hey uh I'm Jawahar, **[01:08]** senior director of engineering at Techon. Techon is an automotive uh retail cloud platform and I lead our uh AIdriven engineering transformation. Right. Starting from defining the person Persona framework to deploying AI agents across development, QA and product. So we kind of started this uh small internal experiment and today more than 1300 engineers uh use AI in their daily workflows. I'm looking forward to sharing all of that uh today and how we made that possible and what meant for **[01:39]** our uh teams primarily. >> Yeah, great. And before we get into it, just tell us a little bit more about how um Techon is kind of uh how it's constructed with the teams. I know that the AI enablement piece has been really across those 1300 um engineers but tell us a little bit more about how you organize um and and the products that you deliver for customers as well. >> Absolutely. So Techon uh has you know primarily three verticals um automotive retail cloud we call it arc and second is our automotive enterprise cloud and **[02:10]** third is our um automotive partner cloud. Automotive retail cloud is primarily the DMS that we have built for our dealership. The beauty is uh the DMS has one uh all integrated pieces together under one platform where the entire dealership operations can be you know run. Automotive enterprise cloud on the other hand you know brings the end consumers who wants to buy a car the car dealer who wants to sell a car and the car you know the OEMs who can who actually manufactures the vehicles right all three entities under one uh platform **[02:43]** completely integrated so that's the beauty the end consumer can buy it the dealer helps out and the you know OEMs kind of manufactures and integrates with the overall platform third is our automotive partner cloud uh primarily enabling third party companies to integrate with our APIs, our open APIs and consumer ton data. So that's how we have actually structured. We have AI enabled you know platform. We have internal ton AI engine to help dealers, consumers and OEMs um you know interact **[03:13]** with Techon ecosystem through AI. So that's one part of it. So that's in in brief. Scott, >> nice. Yeah, useful useful just to know before we get into the detail because one of the big things that I've really learned over the last few weeks talking to you is just like the the enablement piece like the way that you've thought really carefully about how you roll these tools out to a wide range of different people across Techon. Um and yeah, it's been one of the kind of more thoughtful AI um strategies that that I've seen over the last year or so. Um, **[03:45]** and we're really going to get into the detail of that, but tell us a little bit more about kind of when AI or generative AI or AI coding tools started to kind of really appear on your offer for you and your teams. >> Absolutely. So, let me share my screen. Um, >> just take you guys through that. >> While um while Jawah is doing that, I'll just tell the attendees we've got live today that there is a Q&A functionality in um Zoom. So if there's any questions you want to ask Jawaha throughout the session, uh, drop them in there and I'll try and pick up as many of those as we **[04:17]** can, make it a bit more interactive. >> Perfect. >> Yeah. So let's talk about the history a little bit, right? This all started back in um you know Q2 2024 um where you know there were flood of AI tools across um the internet and we started uh evaluating a few tools. What is the objective? Objective is to give the power to our developers product QA and across the board so they can utilize the **[04:49]** power to deliver the innovation that we always deliver right we are a very exponentially growing company uh and we are very limited with bandwidth. So we wanted to give more bandwidth to our engineers to think ask right questions uh but then AI do the grind right so we evaluated a lot of tools all top tools at that point in time so 3.5 was the LLM at you know those days which powered the the tools that were available we evaluated um some tools excelled in you know coding a little bit uh but most **[05:20]** of the time we found for unit test cases AI really excelled well that was not enough uh we started continued the P uh you know back in May 2025 this year um I personally started implementing a large you know feature when I implemented that feature I noticed uh using one of the very popular ideides uh with Sonnet 4 of course I started implementing the feature it kind of built it but then I also noticed that as a human we missed you know a few **[05:50]** features I missed it test cases were not written and whatn not and I tried to incrementally add those features uh uh to the code. I found that u you know the IDE and the LLM kind of hallucinated a lot and it went and broke the you know code pieces and feature that was already built. Then I had to prompt again prompt again to make so many corrections. Um at that point in time I figured okay there is a method to this madness and that is when the person-driven AI framework was born. It was completely inbuilt very **[06:21]** structured framework which we'll talk about in the session. At that point in time again we figured augment code with a context engine uh that augment has built. It's so powerful it helps us a lot and we did a PC and we were successfully able to use the personal driven AI framework on augment context engine which amplified the uh the result that we actually got from the quality perspective. Come June 2025 we rolled out augment code across the company. More than 1300 engineers started using it. June and August 2025 we did a lot of **[06:52]** onboarding you know sessions adoption uh you know goal set uh by our CEO Benu CTO Benu Matthew and the persona AI adoption you know started kicking off we noticed in some of the pilot teams that uh you know started adopting at that that point in time they picked one Jira two Jiraas and then 50% of the Jira tickets were implemented using the framework we noticed 50 to 85% productivity gain just to give you a picture right if you use a raw LLM calling the API and started you know start building features you could **[07:22]** expect somewhere between 30 to 40% you know gain in you know productivity numbers with the persona uh framework we achieved today 80 to 100% productivity gain I'm not exaggerating at all come September 2025 we took another step we made code reviewer AI we built it and implemented part of integrated part of our you know gitlab pipeline so today every MR that are that's being raised gets is automatically L1 reviewed by our code reviewer AI. We built unit testing **[07:53]** AI you know unit testing bring the unit testing coverage from any number that you have to 90% plus coverage within few days completely using AI and uh today more than 20% of our codebase uh you know coverage is we cross 90% plus >> yeah you >> just to yeah just to kind of sum up so you were sat there in May kind of playing around with Sonet 4 and you were just like this isn't quite what we need **[08:24]** it to do at an enterprise level. Um, so you kind of you discovered yourself that if this really going to work, you needed it to be persona driven. You needed it to know understand the context of various different personas across the organization. And then is that what led you to augment rather than it the other way around where augment kind of sold you on the benefits of the persona driven? >> Absolutely. >> Yeah. Yeah. >> Absolutely. >> And solid 4.5 and GPDI again amplified a lot. So one thing that's very important for all of us to understand is any AI **[08:55]** framework we should never target 100% accuracy that's not going to happen AI will hallucinate and we need to embrace that quality and hallucination is good in many cases right the the framework that we build or you know that any of you will target around 80% accuracy through strong prompts through strong context and powerful context engines like you know augment or you know others right together targeting 80% plus And of course remaining 20% we should leave it to uh the LLMs to solve right GPD 5 and **[09:27]** sonnet 4.5 you know they are really good and they amplified our quality of output and our current personas to more than uh you know that we could imagine come October 2025 uh some of our teams achieved 65% to 100% productivity gains right so that that's our history in brief >> so Joha Um, you know, we were we were chatting a little bit before this. Um, and you know, I think some people might **[09:58]** be interested in how did you make the leap from an experience I think we've all had where it feels like we're fighting the model. You ask for one thing, it breaks the last thing, and then you ended up with this personadriven framework that I know we're going to learn more about, but sounds like highly specialized individual agents. Um, you know, I think you'd mentioned something about, you know, observing that the agents were good when when given a set of tasks or a task list. Is that part of what helped you **[10:29]** bridge from you know that technique that we've all started with to the much more specialized framework? >> Yeah. So I can dig a deep dive into that. So when I taking back to the same example right when I started building that large feature um it's like you know I had to go prompt multiple times. My entire day was about prompting. There's just too many prompt hundreds of prompts. That's not how we can do on a day-to-day basis, right? You're not going to gain productivity by doing just that. Maybe **[11:00]** little. Um, so going backwards, how do we make AI or LMS less hallucinate, right? So maybe break down the task and, you know, supply to our AIS do the do just that. So a large task broken down into smaller subtasks and asking our AIS to do uh the implementation works out much better, right? It it has worked really really well. But then for you to break the tasks, you need a help of an architect, right? If it's a complex **[11:31]** problem or a medium complex problem, you need an architect to solve a requirement, design it for you, call out the functional non-functional aspects of the problem and uh then give it give that solution to the engineering teams in this case AI teams to go break down those tasks and then supply to the to the AIs to do the job. The beauty is breaking down this task is part of the framework which is again done by AI agent. The architecture that I just spoke about is **[12:02]** again done by AI agent. And how does the architecture uh you know get built by the AI is through the requirements. So you write a solid requirement with acceptance criteras and all the edge cases dependency called out architect can do a better job. So we ended up building a requirements AI. So given a very high level abstract requirement it can take the abstract requirement find out dependencies and you know you know functional non-functional aspects of it based on the past context that's where again augment context region comes into picture right along with the generated **[12:33]** context this these AIs play a vital role in every step of the way but by the way remember that I spoke about AI being 80% accurate it is not 100%. So human in the loop is very important. You do requirements AI give a Jira comment the human gets in reviews it says yes no and proceeds to the next step. Better better still when when you invoke requirements AI AI stop there saying hey these are my **[13:04]** assumptions I have these many questions can you please help me answer right help with the answer. the human jumps in and answers and clarifies and then the AI generates it. So that's even much better. Right. So that's how we have actually implemented it and towards the end you see the productivity gains that you're looking at. >> Mhm. >> Yeah. And Joe, which persona did you start with and then how did that kind of become what you now know of as the frame the persona framework that you built out at tech? **[13:35]** >> Okay, let me just go to the next slide here. You see what I'm talking about? Uh this is the entire flow that we have today. >> Mhm. >> I spoke about going from backwards. That's how it was built. But if you look from the left to the right, starting from a Jira ticket, when a Jira ticket is created, you invoke requirements AI. It's primarily a product persona, right? A Jira ticket with a very abstract content, abstract requirement. Invoking a requirements AI converts the JATE into a detailed Gatic **[14:08]** with acceptance criteria and the dependencies and whatn not based on the context that is available and once the requirements are done the human review uh you know comes into place the human jumps in reviews the requirement and then says I'm good move to the architect AI. So the human again as a developer you invoke architect AI in your IDE and the architect AI takes the requirements designs the architecture for you based on the current repository current system and the context that is available with **[14:38]** you generates the architecture. Then comes your planner AI where the engineering manager or engineering team picks the architecture jeratica and the feature requirements uses planner AI to break down the tasks into multiple subtasks. Again, engineering manager or engineering team reviews the subtasks and all the jet tickets gets created automatically by the AI by the way. So there's no human intervention here. The AI agents create these tickets. >> Human just reviews it. **[15:09]** >> The developer AI in this case multiple different developers can use the developer AI to build each of these datas one by one. Again it you know creates a pull request in GitLab or any Git system that you use automatically. Our code reviewer once the pull request is created automatically kicks in and reviews every pull request that was created and leaves all the review comments there. How do we ensure if all of this is right? Checklist AI comes into picture. So every persona here has **[15:39]** a checklist inbuilt and I'll show you one of the sample checklist later. It reviews uh every step every person's output based on a certain checklist. It scores based on the mandatory and you know optional you know sections and say say this is my overall score and if the score is within a threshold it go to the next step. So technically if the checklist doesn't happen you know doesn't exist what happens the human manually reviews it but we kind of capture everything in the checklist the human just have to review the checklist output and then move forward better yet **[16:11]** with our headless system that we are going to do today currently building even this is automated so basically another AI reviews all of these output and then takes a decision on whether to move to the next step or not. Same applies to the test developer and test designer AI that you see below. Our QA teams or developer teams who who does QA themselves, blackbox texting, they can use these AIS test designer, test developer and test executor to generate test cases and test uh scripts and check and you know there uh into the **[16:42]** repository and then final approval. Just to give you a perspective again on the left if you see it down this is our prompt versus if you don't have a framework you know defined for yourself you will end up doing hey can you just go implement this you'll do corrections over corrections over corrections so much typing so much talking to the AI and you do it but here we just say as developer AI implement this ticket that's it the detail prompt and what to pull what context needs to be pulled context in the sense here examples are coding guidelines your design guidines **[17:14]** guidelines, your libraries that you have or frameworks that you have in house that you wanted to use part of your you know development all that are by default implemented part of the framework. So just one prompt is good enough that will go and implement everything for you. So that >> yeah do do you find engineers now start to have this like bank of like oneline prompts just like kind of like sat on their desktop and they can just pull from them as they need to and then it will become second nature over time. Right. >> Absolutely. **[17:44]** Um, Jo, could you show us that that checklist? I think I'd love to see kind of like what the steps are that you're kind of checking against and and where a human might want to step in. And I definitely think it's fascinating to think that you will then have a, you know, eventually have an AI doing the checklist as well. And you're just going to kind of move the human further left or right. Matt, I don't know. I I keep forgetting whether we're shifting left or shifting right with this. >> Absolutely. I think it's difficult to to to place it in a way because this is such an advancement of how we're we're **[18:16]** thinking about the the software life cycle. Certainly, I think uh it almost doesn't feel like a shift left or right. I think what's so impressive about this, it's thinking about how can you use agents at every part of the the software life cycle rather than just inserting something in one place as a shift left speed up X. really we're looking at the entire landscape, but I I I really love that, you know, Joa is still talking about human in in the loop. I think it's **[18:47]** great to be, you know, really ambitious and putting these these agents throughout the life cycle, but it's also really important to be realistic, right? We haven't had a super intelligence moment with these models yet. So, designing this framework with the human in mind is, I think, what's really leading to the success of it. >> Absolutely. Yes. Yeah, Jar, talk us through this checklist. >> So, you have one of these for each um >> Yeah, we have one of these for each persona. Yes. Uh definitely needed. **[19:18]** >> Um you're you're looking at architect AI checklist right now. What do we have here at a high level? We have a mandatory checks and we have optional checks. >> Mhm. >> And end of end of the day, uh the architect generates a system design. So we validate the system design generated by the architect part of the checklist and every file you know this file what all sections does it should it have for example an architect uh you know system design should have HLD LLD the flows and **[19:48]** change impact analysis implementation guidelines and whatnot backend architecture if it's a backend you know specific thing you know what are the backend related check that we have to do and front end related checks diagram diagrams in this case we use mermaid and the mermaid diagrams you know and the sequence flows and you know whatnot right all these are defined currently part of the checklist now if I quickly scroll up I I'll just take one example here so **[20:18]** for any AI as I as you said context is very important the first step it's a we have something called as a context builder AI which kind of crawls the entire code and generates you know MD files in textual format with designs and whatnot our context builder AI generates the context for the repository so that LLM you can just spoon feed the LLMs you know to do it once you generate the context and you know inserted part of the framework what we do is uh when the developer calls the prompt the context loading happens first and once you supply the context to augment and **[20:48]** augment takes care of the rest right so if you see here um evidence log investigation JSON created with queries hits decisions they are very specific to you know teon you could say minimum 5 to8 targeted architectural investigation queries executed using codebased retrieval. New components requiring created creation documented. So any new components that you you create uh documentation exist or not highle design completed for the chain scope or not. Um **[21:19]** so a feature could be a new feature built or an incremental feature. So for any incremental feature which is the chain scope, have you gone and updated the HLD? has the architect a updated the LLD right u so all of that and towards the end architect architect a stops for a human uh you know review and it takes hey approved or not is a revision needed or not did the AI do that or not so why are we even doing this check is because AI is hilenate you know there's no **[21:50]** guarantee that you put anything in the prompt get gets executed right that's a creativity part of the LMS itself which is needed but the con is even though you reduce the temperature to as much as possible to make it grounded a is missing. So this checklist ensures uh you know everything is captured as per you know what we've defined towards the end we have the validation summary what's the scoring mechanism that we have what we what the checklist AI does is it fills the checklist gives out the score I'm going to show the score you **[22:21]** know when you go to the dashboard but you'll see the scores there that got generated and it'll be it's very impressive whenever it goes below the threshold that's when we get notified and say okay what really happened with that and how do we go help and you fix the uh the framework itself for example, right? When any of these doesn't happen, it'll automatically fail. Right? So this is a sample checklist to score. >> Perfect. Um and then so this is like a real this feels like a real change to how an engineer might go about their **[22:52]** work to me like they now are kind of doing more overseeing. They're kind of doing more checking obviously with the checklist here. Um, how did you start to bring everyone along on that journey um with the with this change towards the the Persona AI kind of framework and did you get resistance? How did you kind of get everyone to to to shift towards this? >> Uh yeah, initially we always have uh things to deliver, right? um you know **[23:22]** teams have did their job you know to deliver features and you know meet the sprint commitments and whatnot and they were a little worried about hey whether AI will do uh you know what is being promised and that was like eight nine months back right >> uh but then we started with a small team my own teams you know small P we started with one Jira ticket then I did have very aggressive goals uh you know had very aggressive goals at that point in time so next sprint we going to do 50% of the tickets using delivery miss it's fine we'll take care of it right so the bold decisions that **[23:54]** we took uh and with the support of our you know leadership was really really helpful and our teams even though little fearful took the challenge and then did it h you know make it happen right so when we implemented the you know first 50% of those giraads using AI this framework we did see you know problems here and there you know especially for the front end right it doesn't really implement as per what's in Figma things go wrong. It doesn't follow the design tokens and whatn not. These kind of challenges were always uh there but we **[24:26]** evolved uh and we saw success with a 50% uh you know geratic implementation even if it's like a 10% productivity improvement it's still great let's just move forward not take a step back we said next sprint you know we'll do 100% uh AI only implementation so my teams really took uh took that challenge and implemented that that validation helped uh really because we took that learning and applied it across the organization. Uh today now even ac across the **[24:57]** organization you know there are uh you know engineers who are little worried about you know asking questions about quality you know all of that but that to me is a nice problem to have because we have already pro you know shown that it it it works and with code review AI it's kind of mandatory running in the pipeline it's anyway proven uh you know really successful. So that problem statement we are right now working on with multiple adoption uh you know sessions and workshops and whatnot. >> Yeah. Tell tell us a little bit more **[25:27]** about that code review AI because um I think that's kind of a key part here isn't it? Code review is so you know part and parcel of what an engineer does every day but it's also can be painful. So I think that yeah you you've kind of really driven hard on this part of the software development life cycle right? >> Yeah I can show you that. Um I'll show you an example directly in the in GitLab itself. Right? If you see this, my architect generated this like you know yesterday. It's like really new. So if you see this you know GitLab bot **[25:58]** token um you know this is you could think of like a you know bot that we built this code reviewer AI bot behind the scenes it uses the same framework same checklist and Auggie CLI right? Right. Augla is a runner in GitLab with a persona present and the you know MR created at this point in time. The GitLab runner executes and executes against this codebase. Now this is the output of the code review array. This is a summary. Now if you go to the detailed changes there are inline comments given **[26:30]** in uh every line. So you'll see inline comments down but let's talk about uh the overall summary here. The summary it it talks about how many issues found, how many are critical, major, minor, security statuses, right? Any PIIs and any other security validations or issues. Uh what are the requirements coverage? Again, the beauty there are a couple of tools that we have evaluated for code review. There are open tools available. These tools check the current code. >> I hope you're enjoying this video, but did you know lead dev has a lot more to **[27:02]** offer? When I think about lead dev, I think about bricks and hearts. Bricks are those foundational things that you need to know. And lead dev provides that through webinars, through videos, through cutting edge research, but hearts are the things that you need to glue yourself to the people at work. And lead does that through conferences and meetups. >> Make sure you check out leadv.com. Good. But do they check against the **[27:34]** requirement that was you know originally created? Right? So we need to evaluate the code against the requirement that was actually created. Our code review code reviewer AI kind of outperformed any uh of the those we kind of evaluated three to four tools. It was you know with Auggie CLA of course um it did a really wonderful job doing that and bringing in the code context and the organizational level guidelines coding guidelines and whatnot. um our code reviewer AI you know did this analysis critical issues you you talk about the **[28:05]** front end issue that you know that was found in this PR it talks about adding null check this is a good one right null check is a very common problem across you know back and front end talks about that it talks about the major issues that were found and you know other minor issues and it talks about the test coverage and whatnot right so engineer >> can look at this and of course Engineers are really worried about the inline comments that's even more useful in line can just go accept it and move forward **[28:36]** and we you know once this is done what does it mean this is good bad you know we have built a tools and dashboard to evaluate that and uh that's how we take a call on it's good good review or a bad review >> yeah I mean I'm not an engineer but this even looks familiar to me like just from a content perspective when I like work in WordPress it's like here's your issues to resolve did it pass did it fail like I can just work through this. It obviously like really simplifies the task, right? >> Yeah, very much. **[29:07]** >> Yeah, I think co code review is one of those things that a lot of engineers don't look forward to, right? And we, you know, we hear common issues like through all organizations when you've got engineers across the globe, you end up with, you know, a long delay. Sometimes the person who needs to review is 8 hours away from the author >> and then you have that sort of painful time between every review and update from the author. And also we've you know **[29:38]** we've heard a lot that you know with more people using AI some PRs are becoming bigger which can result in people rubber stamping right they it's too much and they're like oh I'll just approve it. So all of this seems like, you know, an incredible first pass way to break down some of the barriers to people even getting started in the in the review process, right? It makes it feel easier and um something that you can, you know, fear a bit less because something's already made a first pass **[30:09]** for you. >> Absolutely. And it just does it immediately, right? You don't have to wait for an engineer to be available to come review your PR, but it just invokes immediately. >> Yeah. And um Javaha, you're tracking this at a higher level, aren't you? In terms of like how effective the code review AI has been. Do you want to show us a little like some of those results? >> So this is the dashboard that again my architect built over the last two weeks is AI is so powerful that we could build even a dashboard you know that fast. Um **[30:40]** so use what you're seeing here is a customuilt uh you know tool uh in-house build and I've selected a May 1st to May 31st uh as a date. This is when we did not have um AI implemented that much uh in in Techon and I also will take you through how it looked in September and you know last month October. Right. Awesome. Now this dashboard if you see here um gives us the statistics on all our repositories that we currently crawled uh talks about how many MRS that are merged back in May 810 merge **[31:13]** requests merged in that particular month. Average time taken to merge an MR 2 days and 9 hours. Average time to first human review 2 days and 24 hours. Total comments uh you know around 947. Total human comments were around 929. At that point in time, we were doing some P with AI. So you see 18 AI comments also because we kind of played around with the personal framework. Average comments total comments resolution rate right **[31:46]** 9472% of those were resolved and 73% of those human plus or minus resolved by human. Now total lines of code delivered 336,000 right so this is the dashboard that we have now fast forwarding to September I'm just going to scroll you know go up and down you see the difference here >> so interesting yeah >> this is the impact of AI this doesn't talk about just code view right if you **[32:17]** look at the number of MRS merged with AI adoption you see that age 10 to 1477 7 right 2 days 8 hours 2 days 3 hours and time to first AI reviews it is 1 day and 3 hours but current AI code reviewer that we have reviews that in like you know 5 to 10 minutes immediately as an L1 reviewer um and it doesn't wait for um you know **[32:48]** the human to come and acknowledge and you know raise that total number of comments here is 10,200 total human comments are 2400 7,788 AI counts right are given. Now what does it mean? If AI is giving 7,700 we need to look at and we'll talk about how many of these are valid invalid in a second but 7700 AI comments uh on across you know take on that saves so many bandwidth so much of **[33:19]** bandwidth of human and gives it back to the company so the engineers can focus on asking right questions and you know use their creativity for you know the regular work not worry about you know spending time on this right that's that's what it means Total comments resolution is around 48%. Total human comments resolved 71. Total AI comments resolved that was back in September 37. And reduction in human comments 46%. So that is one of the very important metric and total lines of code although **[33:51]** it's not a primary metric. I'm just giving you as a as an example. 453 right? It went up. >> Nice. >> Let me go to October you know for a second but you see this you see this here like average time to first AI review is 2.1 hours even this is high because there's a CI/CD pipeline runs then you know our AI code review runs so it's because of that otherwise it's just immediate right total MRS merge total comments you're looking at total AI comments at 7200 and total AI comments **[34:22]** resolved is 35%. Now I want to say one thing right we're not going to sit here and just keep watching this dashboard manually. We are now implementing something called as the AI autolearning. Based on this 35% we are going to figure out why 65% of those remaining comments were not accepted. We want to find the behavior patterns of what AI uh comments where bucket them and figure out how we can maybe are those valid comments or invalid comments. Invalid comments in **[34:54]** the sense we going to take it as a guideline and make AI not repeat the same things. But if it's valid comments we are going to put it part of the you know guidelines and we're going to talk to the developers to see why they were not able to accept it. On the other side 73 of% of those human comments were accepted. We're going to ask question about why a couldn't find it. why humans still did uh you know these many comments in the PR like 1800 even though it's very less find the behavior patterns good patterns implement that **[35:25]** part of the you know code reviewer guideline that way AI finds it the human bandwidth is reduced so give it back to the company so we going to really shift left not just this once we figure this out we're going to even ask this question right why AI is finding so many comments >> should the previous personas implement it in the right way so that it doesn't come to the code reviewer in the first place. So that's how we we are planning to autolearn and then shift left and improve our personal framework overall. Think about >> it's a new yeah it's a new interesting challenge that you didn't have six **[35:56]** months ago. Yeah know if if you implement this u your AI for AI's vision that we have comes into you know play because you shifted left now these personas are like you know more of an execution personas really have nothing to do now AI can be as autonomous as possible of course keeping human in the loop but the accuracy goes really really high more than 80%. So that's our target goal. >> I'm I'm curious, Joh, you know, I I would say this is a very sophisticated **[36:28]** and mature thing that you have at this point. Obviously, it sounds like you're going to iterate a little bit more, but um for anyone who's listening in, kind of sounds like code review might be a good place to start, right? If you haven't done any persona yet, you get going with code review. And which of those metrics do you you think is like most interesting? Like if you can just measure one or you haven't reached the point of maturity of having a whole dashboard yet like what's the what do you think is like the core **[36:59]** signal that leaders can can look at to understand that code review is is having a good effect. If you ask me um you know I'll just go back to this dashboard most important metric uh I would say is the AI comments that are accepted by the developer right end of the day it's the developer who accepts today >> uh and if you see 70 to 100 AI comments and 35% of those were accepted >> right and now we need to increase this **[37:30]** percentage and reduce the human comments p you know percentage if you see here this is fine but we need to reduce the number of comments here right so once we have the person are strong enough if you are able to achieve this and I think this is the ultimate metric from code review perspective and second the quality of the code review itself the checklist that you're going to the persona that you're going to build the quality of that person also is important which we'll talk about in in a second right let me just go down here **[38:05]** So this one u you know this dashboard we have is for all the personas that we have it has quality scores how many times it got executed and what is the quality score primary here right for code review per se uh we have something like this if you see here in the last 30 days we had 13,900 reviews done by our code reviewer AI on an average the quality score is 94.69%. This score is important, right? It ran against 2700 Jira tickets **[38:38]** across seven users and you know 39 competence. Seven users because it's it's bot user one bot runs everything right. So this quality score if it goes down uh you know less than 94 95% we just need to double click and see if uh you know checklist or persona you know what you need to improve to uh achieve that score. And second is this human comments and total AI comments and ultimately of course how much bandwidth we are going to give it back to the developer to the company so that **[39:08]** the engineers can focus on something else incre increasing the overall story points or velocity of the company. Yeah, that looks that looks great. Joha, um I think it's really interesting to see all of this telemetry. Um because a lot of the a lot of the stories that we we hear are often a little bit finger in the wind. Like we're using AI and we we know it must be having some kind of effect, but we're not necessarily sure **[39:39]** what. We don't know how to measure it. Um, you know, I think developer productivity is is one of those things that challenges a lot of organizations for how to get a a handle on it and what to look at. So, this is a you know really interesting perspective of you know how someone who's who's leading the charge in terms of a like a personadriven framework is thinking about measuring. Yeah, we've done a bunch of research on this throughout the year and we've got a new report coming out **[40:10]** uh this week um where we actually asked you know are you measuring the performance of your AI systems or LLMs and it was about a quarter it was about 20 to 25% so it really is a it really is the minority at the moment who are doing anything at all and what Jawah is doing here is not the bare minimum we can see that it's you know telemetry right the way across all the personas with kind of this deep dive into um into code review and I love that this is all homegrown in what six months show is **[40:42]** that right this tool yeah yeah overall personal framework yes last six months >> yeah cool was there anything else you wanted to show us on the um analytics side or telemetry side >> I can show you this personas for example this is a heat map uh that again our team built in the the last few days is we have all these personas. You can keep adding more and more personas. We have a migration persona. Migration is one of the common problem that I want to just bring it out, right? I have written code in Python. I want to convert to Java. My **[41:14]** uh codebase is legacy. I want to convert to a, you know, new codebase with all the nice, you know, bells and whistles and whatnot, right? So, migration personas is something we do. And, you know, all these personas telemetry is important. You know, you cannot improve what we cannot measure, right? What we don't measure. So it's very very important to add telemetry to each of these personas. It's very simple. You have a large you know prompt in your persona. You have your telemetry uh calls made and you you track it in your external system. It's as simple as **[41:44]** that. Um this talks about you know day-to-day basis. Um these personas were invoke how many times and across you know different dates right? So this kind of gives us a visibility on adoption that we want to make sure all these personas are used across the company to give give that productivity gain. So this is this kind of helps and we reach out to teams you know we have task force created uh you know we work with the you know the teams in tech on building jiraas one by one we **[42:14]** take the feedback we go fix it and then we again go back to the teams and from adoption perspective right so and then we look at the score anything 94 96 100 great if it goes to 70 82 uh we definitely look into it right we'll see you know why it is done this way how do we improve the prompt if you improve the prompt you need to have eval framework to test it. So we we have built our eval framework in NS and that's something we you know constantly keep you know improving and continuously keep doing it. This dashboard also **[42:45]** >> team level analysis and whatnot. Yeah. >> Sorry J we actually had a question from Lauren who's who's watching asking um how you measure that quality score um how how what is considered quality? >> Yeah so quality score again going back to the checklist. This checklist has both completeness and quality uh you know measurements right every persona gives out the output and whether the output that is generated is as per the requirement and as per the architecture that was already generated right so it kind of compares based on this checklist **[43:16]** the checklist is very custom made for each of the personas >> finally uh you know you see here there's a pass fail criteria how many mandatory passed optional passed you know what's the score and what's the overall score this overall score is what you see here in the dashboard as long as your checklist covers the quality criteria part of that the quality score can be computed again there's no manual work you are you have LLM as a judge here right you use LLM as a judge and compute the score here **[43:48]** yeah and of course you know the developers go and accept it merge the PR it goes you can obviously track those uh and you know added part of this today we don't have it but definitely that's something you can add additionally If the mer the MR gets merged that again adds to the scope. >> Got it. >> Is is there something that gave you confidence to use an LLM as a judge? Joa certainly like we're seeing that be done in in multiple places but I know some people on on first hearing that they can **[44:21]** be a little bit like okay I'm using an LLM to judge the output of another LLM. like am I just stacking things here? Um, we've we've certainly seen it be effective, but I'm I'm just curious as to your take on that. >> Oh, very much effective. By the way, you can use the same LLM as a judge, too. Uh, we pretty much use do that. And you can also use other LLMs as a judge. Um, you know, there are different patterns, right? You can have a challenging challenger pattern. You can do a self-critique pattern. For example, some **[44:51]** of our AIPC architect and developer, right? developer implements it, architect, you know, designs it, it critics itself, self-critique and if there's a problem with it, you know, it'll do one more iteration and then passes it. It kind of improves the quality of output. Now, you can have another agent primarily the checklist AI is another agent for example or you know a debate AI or a challenger you can have the other AI question this you know AI's output and you know take it forward from there. So that really helps. Uh **[45:23]** definitely using two different LLMs, two different model also, you know, helps helps a lot. >> Yeah. Yeah. We did a little bit of research that found that generally when we used an LLM as a judge, it usually agreed with what a human would would make judging the the same situation. And so it it seems like a really a great strategy to be able to, you know, in this case judge quality. >> Yeah, absolutely. >> Nice. Um Joah, we're running out of uh time and I really want to talk a little bit more about what you're working on **[45:54]** next. Um you've made so much progress here, but I'm sure there's more personas in the pipeline. There's uh more tweaks you can make and and definitely more things you can do to kind of keep on this AI journey across the company. >> Sure. So if you see from development perspective, we are enhancing the developer even more uh primarily around the front end aspects, right? How do we have a tighter integration with Figma so that just the Figma input given uh the developer AI implements the feature ticket. Today we **[46:26]** take screenshots. We do a mapping with the Figma components uh with our code bases. We generate a context out of the Figma design system into a you know store it in our codebase. We invoke the developer AI which uses the context that we generate. Again augmented context engine powers that and then we build it. It works great. Uh but it can be done you much more better with a tighter integration with Figma. Uh then you have the code review. I spoke about autole learning measuring human effective hours. How much hours we are giving back and reducing code review issues through **[46:56]** you know shift lift left approach that we spoke about that integrated with Jira workflow. This is very important. Now today we can use augment ID the the ID extension to invoke all of these 13 different prompts that I you know just spoke about. But what if it is integrated with J itself? Now you have a ticket. You can assign the ticket to an AI agent. In this case, architect AI. The architect AI once it's assigned and the state is changed. It automatically picks the ticket, does the architecture, attaches the content back to the jitter, **[47:28]** invokes through a slack, you know, informs a human via Slack or email. Uh the engineer can or architect can come in and look at it. Hey, it looks good, not looks good. you know we can then you know pitch in via other mediums to actually correct the architecture that is something we are looking at uh we are currently working on the security AI you know take the threat modeling and you know uh PIIs PIS are already implemented but other threat modeling and security aspects building security AI integrating it part of the GitLab pipeline and of **[47:59]** course the product management personas right starting from your BRD you know you get to PRD and create stories and then your regular personas you know kick in Right. So that is something we are working on. We are working on the headless uh orchestration um AI managing AIS. Um you know think of the the situation where you have scrum team of AIS only. So right there's an AI scrum team eight or nine agents with a you know manager agent you know it automatically picks a bug. It automatically picks a feature implements **[48:29]** that and of course at any point in time you know invokes a manager AI LLM as a judge reviews the output of each and every persona when needed 20% of the cases invokes a code reviewers uh you know human code reviewers and human code reviewers reviews the output they can go fix the bugs do the anomaly detection automatically try to fix the bug does the RCA raise a pull request invokes the human for review and whatnot That's something really **[49:00]** working on. >> Amazing. Lots to do. Um, so if anyone's got any last questions for Joah before we get out of here, please uh get them in now. But Matt, is there anything you wanted to kind of ask at the at the end here? Um yeah definitely some recommendations that I have uh for leaders of um there are a lot of tools in the internet **[49:31]** available you know as you said but then later we now try to work out if that's Jamaha >> yeah I think uh Yeah, >> it is. >> That is very unusual because it's normally it's normally me. So, I was very nervous that it was me. Uh, it's even worse. It's the person that we all want to hear from. So, >> um, let's give it a couple of sec here **[50:03]** and see if he jumps back in. Hey, Joa. >> Hey, sorry. Not sure. >> That's okay. You look like you're back. >> Okay. Are you able to see my screen? >> Yeah. Um, so let me just kind of catch us up where we started to lose you there. So I think you were starting to give some advice to folks who are maybe uh starting out on this journey too. >> Absolutely. Just please give me a second. I just lost the slides now. **[50:33]** >> Yeah, no stress. >> Okay, able to see my screen, right? >> Yeah, there you go. >> Cool. So um there are so many tools uh available in the internet, right? Um you know we can use all of those tools. they can give incremental you know productivity gains in various you know steps of the you know PDLC or you know your day-to-day but starting with structure is very important you first define the structure end to end what we trying to achieve then try to fit the tool or your custom you know your own framework no problem if you don't do **[51:05]** that we'll end up building or you know you know procuring so many tools and have a stack but later to figure out none of these talk to each other right because end of that's what we want Right? We want all of these to talk to each other pass the context so that we have one system uh and we are able to achieve the Uber goal product managers talking to engineers and and QA and whatnot the same concept making AI a first class team member uh you know giving it a name giving it a scope giving it an accountability one AI **[51:35]** cannot do too many things you know define an AI can only do so many things and if it's a scope creep go to the next agent and next agent and whatnot it should be treated like a developer in your organization and with clear metrics uh you know and accountability baking AI into the process not as an experiment so integrate it part of the workflows you know be a PR workflows be it Jira or you know day-to-day uh not uh as something that you can also do because that that system won't work **[52:06]** because we usually won't do that right it needs to be part of the system part of your you know process balance autonomy with guardrails use checklists very important That's the only way we will have visibility otherwise a can do anything that you know it'll do you will not have visibility at all checklist human in the loop validation and compliance layers are very very you know critical that way we can build trust on the ASA you see a score 94% you can say hey I trust you if we don't have a score there's no trust **[52:36]** focusing on enablement over enforcement um you know upskilling people you know how to write prompts how do you you know learn a you know give them all the tools tools and techniques and learnings as possible. But then use the framework because with the framework you get the advantage of defining the overall governance rules, guidelines, you know, checklists and you know design guidelines, coding guidelines and whatnot which earlier we left it to the human to follow. Although we try to create plugins and whatn not today with the framework you can enforce that at **[53:08]** the you know at or level it's very powerful and that way upskilling people and making them learn and adopt is more important than just enforcing it measure everything checklist measures metrics keep monitoring and evolve from AI assisted to AI orchestrated you can have agents invoked via IDE but then automated agents you know through orchestration is the way to go and tomorrow you know you 6 months or one year down the line I could only imagine AI is invoking these orchestr **[53:38]** orchestrations automatically so that's going to be the super powerful you know next so these are my recommendations >> yeah nice and what would be the one thing that you know now that you wish you'd known when you started out here >> um I think u creating a framework um and I I think adoption uh I would have thought I never thought it would have been so difficult. Adoption is very difficult. **[54:08]** >> Yeah. >> Uh changing you know developers and engineers mindset to say hey use the framework uh is hard but then um something differently I would have done uh could be you know integrating that part of the Jira. That's one of the learning gear right integrated part of the system. So the developers get to use it you know on a day-to-day basis you know part of the process. Nice. Um, it looks like we got no more questions. Matt, have you got any final thoughts or final questions for Jo **[54:38]** before we get out of here? >> I don't know that I have any final questions. I think this has been like really really informative. Um, so can't thank you enough Joha for, you know, being willing to to join and share all of all of this knowledge with us. Um, I feel like I've I've learned listening here. So, um, hope it's been the same for everyone tuning in as well. >> Thanks. >> Yeah, totally. >> Yeah, totally echo that. Um, thank you, Joahar. And, um, I think I'll just take **[55:09]** this opportunity to just thank everyone that has been involved in this uh, series. This is our final session, but um, right back to the first one we did with um, with James up to today. I've learned so much about how uh these various companies are actually using AI in action and they've been really generous to show us kind of some of those workflows, show us their dashboards, show us their IDE, show us the stuff they're doing. Uh that was really the idea of this series and I think that we've delivered on that. So, um thank you to all of our speakers for for sharing. Thank you to all men for **[55:40]** supporting this series for for bringing these great people to to our audience. Thank you to anyone who dialed in to every one of these sessions. I know you're out there. Uh, and thanks Matt to uh for co-hosting these with me. It's been it's been a great pleasure. >> Thank you. Thank you Scott. You've you've led the whole thing and it's just been a pleasure to be a part of it. >> Awesome. Well, maybe we'll do some more in the future. Uh to see if we can get some more of these stories out there. But for now, um thank you so much and goodbye. >> Goodbye. Thank you. Thanks, Scott. **[56:10]** Thanks, Matt.