Effective Verification for Humans and Agents ![rw-book-cover](https://img.youtube.com/vi/_FSEkCBJZls/maxresdefault.jpg) 🔗 **Source:** [Effective Verification for Humans and Agents](https://www.youtube.com/watch?v=_FSEkCBJZls) > [!note]+ Document Notes > --- > tags: [ai-code-review, agentic-coding, code-verification, developer-tools, engineering-leadership] > author: [MISSING] > source: LeadDev > reference: https://www.youtube.com/watch?v=_FSEkCBJZls > date created: 2026-08-13 09:00 > --- > > [!info] Generated analysis — not the author's words. Sections below are machine-written > > from the captured source. Only text inside `> [!quote]` blocks is verbatim from the author. > > Do not quote or attribute any other line to the named author. > # 🏷️ Title: Effective verification for humans and agents > ## 🧭 Table of Contents > - [📝 Summary](#-summary) > - [📖 Main Content](#-main-content) > - [🗒️ Notes](#️-notes) > - [📚 References](#-references) > - [🗣️ Paraphrase](#️-paraphrase) > - [📰 Newsletter](#-newsletter) > - [✅ Action Items](#-action-items) > - [🧪 Hypotheses](#-hypotheses) > - [📊 Methodology](#-methodology) > - [❓ Questions](#-questions) > - [🔮 Future Research](#-future-research) > - [💡 Implications](#-implications) > - [🧠 Learning Styles Assessment](#-learning-styles-assessment) > - [📊 Accessibility Assessment](#-accessibility-assessment) > - [🌍 DEIB Assessment](#-deib-assessment) > - [✅ Logical Validity](#-logical-validity) > - [🛡️ DARVO Analysis](#-darvo-analysis) > - [🎯 Conclusion](#-conclusion) > ### 🔗 References > - [LeadDev](https://leaddev.com) — event host, panel series run in partnership with CodeRabbit > - [CodeRabbit](https://coderabbit.ai) — AI code review tool, panel co-sponsor, David's employer > - Amazon customer-service chatbot escalation story (referenced, not linked, ~33:08) > - Capital One AI rollback story (referenced, not linked, ~40:15) > ## 📝 Summary > - **What happened.** A LeadDev-hosted, CodeRabbit-sponsored panel (host Jen Riggins, with Erin — agile coach/delivery lead, Martin Davidson — CTO of a two-person consultancy, and David — VP of AI at CodeRabbit) discussed how code review is changing as AI coding agents take over more of the writing. A live poll of the audience found reviewing-code time roughly doubling over two years (5% of respondents spent 50%+ of their time reviewing two years ago; 23% do now), while AI-assisted code adoption is already near-universal (44% "all of it," 0% "none"). > - **Why it matters.** The panel argues code review's old purpose — catching human mistakes line by line — no longer maps cleanly onto AI-generated code, which fails differently (structurally, architecturally, in scope creep) rather than through typos or logic slips. Teams that don't redesign verification around this shift either drown in PR volume or under-verify. > - **Key stakeholders.** Individual contributors and reviewers (accountability for merged code); engineering managers newly acting as "second-line managers" of agent fleets; legacy/regulated organizations (insurance, banking) constrained from moving as fast as greenfield teams; end users affected by unreviewed bias or errors in AI-authored systems. > - **Risks and tradeoffs.** Loss of the "why" behind decisions when no human reads every line; erosion of the puzzle-solving satisfaction that draws people into engineering; diffused accountability ("who's in charge?"); tacit organizational knowledge (the "Brenda and her secret spreadsheet" problem) that agents can't see; unequal ability to benefit from agentic coding depending on codebase age/regulatory constraints; possible bias baked into AI training data going unreviewed by non-majority engineers. > - **Bottom line.** The panel's converging answer is not "review less" but "review differently" — build oracles (tests, fuzzers, mutation/property testing, deterministic guardrails from strongly-typed languages), read agent-authored natural-language summaries and journals instead of diffs, and keep a single accountable human signer for anything that ships, even when that human never reads a line of code. > ## 📖 Main Content > - The panel frames its core tension early: writing code was never the bottleneck, but reviewing it has become one as AI agents generate more code, faster, in larger PRs. > - Martin Davidson represents the extreme case: he writes zero code and reviews zero code personally, describing himself as having moved from "first-line manager" to "second-line manager" of a fleet of coding agents that review each other overnight and report a summary each morning. > - His replacement for traditional code review is what he calls "Oracle Driven Development" — establishing an authoritative (if imperfect) source of truth (existing standards, reference implementations, performance benchmarks) before any code is written, then progressively handing off trust to agents as design solidifies, with multiple agents running adversarial reviews against each other and against the oracle. > - David reframes "AI-assisted" as a near-meaningless umbrella term spanning simple autocomplete through full spec-to-code handoff, and questions whether the "everyone is fully agentic now" narrative is real or a Silicon Valley-bubble effect — also raising a real infrastructure constraint (GPU/energy capacity) on universal adoption. > - Erin brings the organizational-realism counterweight: teams inside large, legacy, heavily regulated systems (she cites banking/insurance) cannot adopt Martin's greenfield approach, creating what she calls a "multiclass system" where only nimble teams capture agentic coding's benefits. > - A recurring theme is that review is shifting from line-level correctness to higher-order verification: mutation testing, property-based testing, fuzz testing, and even MC/DC (modified condition/decision coverage) testing historically reserved for safety-critical software (aircraft control systems) are becoming ordinary tools once agents are cheap enough to run them constantly. > - Strongly-typed, memory-safe languages (Rust specifically) are framed as a force multiplier for agent trust: language-level guarantees eliminate whole bug classes agents might otherwise introduce, functioning as deterministic guardrails alongside probabilistic AI review. > - Agents "journaling" their own reasoning (an explicit echo of a paper-notebook habit Martin taught junior engineers) is presented as valuable for two reasons: writing forces an extra reasoning pass that catches self-identified errors, and it leaves a persistent, cross-machine record that per-machine session logs don't provide. > - David's concrete catch story — an agent silently adding an unrequested middle tier to a system during a 40-hour unattended run — is the panel's strongest evidence that architectural/structural review still matters, but that it can happen through a natural-language "what happened and why" summary rather than a code diff. > - Accountability is treated as unresolved industry-wide. Concrete practices offered: nothing reaches a live environment without a human sign-off (Erin's insurance employer); the accountable individual is whoever put the change in, even at heavy agentic-adoption companies (David, CodeRabbit); a shared company policy quoted verbatim from the chat: "Own the output... AI made it. You own it." A chat participant (Michael) flags that most teams haven't yet been tested by a real accountability failure. > - The panel closes on human-side costs and open questions: whether new engineers can develop "taste" and judgment without ever reading code closely; whether removing line-level review erodes the puzzle-solving motivation that drew many people into the field; and Erin's concern about representational bias in tools built predominantly by one demographic, surfacing in downstream product harms nobody explicitly reviewed for. > ### 🗒️ Notes > #### ⏱️ [00:01:35](https://www.youtube.com/watch?v=_FSEkCBJZls?t=95) Live poll: review-time and AI-assist trends > - Two years ago: ~33% of respondents spent 10% or less of their job reviewing code; ~33% spent 20–30%; only 5% spent 50%+. > - Now: 28% spend 10% or less; ~33% spend 20–30%; 14% spend 40%; 23% spend 50%+ (up from 5%) — roughly a 4.5x jump in the "half my job is review" bracket. > - AI-assisted code adoption: 44% say "all of it," 0% say "none," over a third say at least 75%, 5% say 50–75%, 13% say less than half. > #### ⏱️ [00:03:18](https://www.youtube.com/watch?v=_FSEkCBJZls?t=198) Panelist introductions — Erin > - Agile coach and delivery lead, Hello, hello, hello everyone and a huge thank you to joining us today for today's panel. This is a lead dev online discussion created in partnership with code rabbit. The discussion will last about 45 minutes after which both myself and the panel will head over to the lead dev slack to continue the conversation in the good old channel of panel Q&A. Now that should have popped up on your screen and then I'll put it in the chat later. It's just a really great community. The lead dev community is a engineering leaders and maybe dev plus engineers just supporting and learning from each other. So, even if it's not that active today in the panel Q&A, there are many, many channels that are active and it's a really great community. I recommend it. But for right now, we also really want to know what you're doing because let's be honest, everybody's going through it right now. And so, we're going to ask that like the chat is on. Feel free to drop in the chat any tips, experiences, or questions you have. So, thank you in advance for that. And then we're going to kick off with a poll right now that we're going to run through because it's three questions of homework. So, it's the most homework I've given on a live stream, but we'll run it while we do introductions and all. So, the questions are, what percentage of your job was spent reviewing code two years ago? And then we go from like 10% or less all the way up to 50% or more. What percentage of your job was well it should be is my typo. What percentage of your job is spent reviewing code right now or this last week per se with the same clues. And then what percentage of your code is AI assisted and yes that is an opaque term. We will get to that because who knows what AI assisted. So yeah, writing code was never the bottleneck, right? But reviewing code seems to be right now. Teams are drowning in PR volume as AI agents generate more and more code, longer lines of code. Uh basically AI was trained in my opinion on a lot of the tech pros that like were over doing presentism and overly confident maybe the more junior the mid the you know the average. So it may not have the best techniques around code quality and for longevity. It's for it's going on what they think you're most likely to accept in the moment and it will say it very confidently. But people want practical strategies from peers to cope with the deluge and review fatigue. So that's what we're talking about today. So I'm Jen Rigggins. I'm everyone's favorite tech journalist and host. And today I'll be joined by Erin, Martin, and David ready to share their experiences and insights. So in that order, let's go. If you want to introduce yourselves, please. Thanks, Jen. Hi, folks. I'm Erin. I'm an agile coach and a delivery lead and I spend most of my time in a room where working actually makes change happen. So, all things ceremonies, making teams like work together, all those good decisions. Um, and alongside that, I'm training as a therapeutic counselor. So, that shapes how I think about coaching teams and leadership through change. And I'm sort of interested in that space where structure and psychology meets. Um so for me on this subject as agentic coding becomes um more and more frequent and starts to reshape what that phrase of team even looks like that's what I keep coming back to and if if we've got AI writing the code what is left for collaboration how do we work together and sort of design it by purpose rather than by accident. >> Fair enough. Those are really all valid points. So thank you Martin. >> Uh hi everyone. I'm Martin Davidson. Um I've been working in software for a very long time. Um I started out working in the the the mid '9s uh on uh Net Meeting, a sort of precursor to uh what we're using today uh video conferencing software over modms. Worked on uh Microsoft Windows for a bit uh Internet Explorer. Um and uh more recently I've really been uh spending a long time building stuff with with AI. um partly uh rebuilding things that I built earlier in my career and discovering that yes, it is entirely possible to build some very large pieces of software um remarkably quickly. Um and then just building uh I'm I'm currently the CTO of a small consultancy and the two of us were obviously building some software which I can't talk about but we're building a whole lot of side projects as well and and you know so if you're familiar with the MP3 codec there's a replacement of it the Opus codec. So, uh, I, uh, did a port of that to Rust earlier this year, uh, which is more performant than the original one. That's all entirely, uh, Claude and Codex's work. Um, I have written a MIDI emulator, if you're familiar with MIDI, um, uh, which, uh, has samples of all the various instruments we have around the house. Um, I've, uh, built an emulator which can simulate PCs back from the 1990s. if you ever done any retro gaming, um it can uh basically do everything DOSBox X can do, but it's about 25% faster um performance-wise. Um we really are in a quite remarkable place, I think, in terms of both how quickly you can build things and the quality that you can get out of the AIS. Um you know, if you're if you're um if you find the right ways to to to to use them. Um so yeah, that that's me. Is it cheating if you want to play a retro game without the pain? >> Uh, no. I don't think so. I mean, I think they should be fun, shouldn't they? >> I guess that's true. Yes. If you want dysentery in the Oregon Trail, you can >> Yeah. >> Thank you so much, David. Please introduce yourself. >> Yeah. Other than playing the Oregon Trail a long time ago, um, my I'm VP of AI Code Rabbit. Um, my background is in AI. I've been the space just over 20 years now. Obviously, it has changed wildly in that period of time and I did not start uh quite where it is right now. Um I was mostly in a math room looking at a whiteboard writing out symbols on on on that for for a long period of time. Um but yeah, I worked in personalization recommendation and optimization problem space at Amazon and Netflix. I started a bunch of companies in various spaces of natural language processing. I ran a generative AI music startup um for eight years uh which served millions of musicians across the world and and helped people become signed artists and things like this. And ultimately that company centers around the idea that AI can be used for human uh enhancement and not used as a replacement technology if you approach it uh with a certain perspective in mind. and I've been at Code Rabbit for just over about a year and a half u at this particular point focused on all things related to how do we leverage this moment as much as possible to make sure that we can build things in really cool ways and build really interesting things while keeping that quality bar high and making sure that we are as being thoughtful as Erin was pointing out about how we're thinking about the future of software development as we move forward. I have to ask you you had a generative AI music startup for eight years. >> Yes, that's correct. >> But generative AI is like a for >> AI the term did not exist. So we so the field of computational creativity has been around since the 1950s and that field has studied the idea of using AI in various forms because AI is not just LLMs. has existed for a very very long time and to create art, to create music, to create dance, to create all kinds of things, books, um, and it's been around for quite a while. And ultimately, uh, my wife is a musician. She really wanted to write music and she was struggling. And so uh we got together and we figured out a way to leverage AI at that time which was mostly LSTMs and and recurrent neural networks and various things like this in order to figure out the structure of music and be able to build systems that would help her to write songs and ultimately that was a paper it was in 2015 and um people started contacting me and asking to use it. I wrote it in R. Uh if people don't know it's a statistics language. It's not very usable or user-friendly, but ultimately it served the purpose at the time. And uh after enough of those emails, we decided to think about starting a company. It took us a couple years to actually do it, but in 2017, we started Wave AI, it was called. And and yeah, I ran that for for eight years uh with my wife. And yeah, it was it was really an amazing experience ultimately helping musicians. Both of us love music. I've been doing it my whole life as as she has as well. And um it was quite an experience building things to service musicians and to make it so that they can realize their their dreams. Right. >> Wow. That's really interesting. And I think you'll bring some unique experience that way too. Although just like Martin, you were in a baby startup at that point which is very different and they are finding more successful. So I would like to end the poll. Thank you. Thank you to our very active audience. So, thank you so much for that. And just to read for those that won't see this when this is recorded, what percentage of your job was spent reviewing code two years ago? About a third of people say it's 10% or less. It was 10% or less. About another third or said 20% or 30%. Only 5% said two years ago more than 50% of their job was reviewing code. Okay, maybe I should have done three years ago, but who knows where we are. Time failed to exist past COVID, right? So now, what percentage of your job is spent reviewing code right now? Another 28% said 10% or less. Uh 20 to 30% is another third. 40% that 14% of the audience has spent 40% of their job is reviewing code. And then going from 5% two years ago to now 23% of the audience says that half at least half of their job is reviewing code. So there's a reason the hot topic keeps coming up. I don't think I I think code reviews are important. I think all of us think are important for different reasons and mentoring is one of them. But I don't think anyone went into tech because they wanted to review code. And finally, what now this is the final question. I'm going to let David explain what he and but the final question is what percentage of your code is AI assisted? 44% of people said all of it. Zero said none of it. I guess you probably wouldn't be as interested in this call if you were. Uh, over a third of the audience said at least 75% of their code is AI assisted. 5% said 15 50 to 75% of the code. And then just 13% of people, but still a significant number, not everyone, said that less than half of their code is AI assisted. So David, this this last question was a bit of a last minute one. What do you mean by AI assisted code? >> Yeah. So I mean it I find this term really interesting and and different companies will use it in different, you know, to mean different things. It could mean that you're using AI like for AI completion. I'm going to complete a piece of code and it's it's boilerplate or whatever else. I'm going to do tab completion. AI is going to fill that in all the way down to I type a spec out and I just hand it off to codeex or clog code or cursor and I just let it run and at the end of it all I'm really doing is testing reviewing verifying what's going on and then iterating on the spec or I'm doing something like that they could go all the way through to spec driven development right and so obviously I I'm I'm in particular in Silicon Valley and people here believe and are trying to purport that everybody is moving to this idea of fully background around agentic coding. And so my curiosity is is that happening uh to people across the the world or is it something that's very Silicon Valley bubble? Um because definitely that's that's a possibility, right? >> And also do we even have the energy GPUs to have everyone jumping into aentic right away. So now Martin, I want to ask specifically you because we had a wonderful planning call with bells and whistles that we've politely politely asked you to turn off for this, but were your agents communicating with you if that's a fair term? So I want to ask you, what percentage of your job was spent reviewing code two years ago? >> Uh I don't know, probably 20% something like that >> as a manager. Yeah. Yeah. >> And now uh what percentage how much code do you review? >> None. >> Absolutely none. >> And now I ask how much code do you write? >> None. I don't write any. Um I I I don't look at the code. I don't interact with the code. Um for a while I felt like I was operating a bit like a first line manager, but now I feel like I'm a second line manager, you know, and and the agents orchestrate the agents for me. Um, and you my my daily sort of routine is I'll get I'll come in in the morning and there'll be a summary of everything that's happened overnight and over the last day. So the each of the members so we've got a fleet of machines and each of the the agents on those will have summarized stuff on per repo because we're working on multiple repos at the same time um and just told me what's been going on and identified the things I need to go and look at. Um so I go and follow my follow my nose on those. Um I I think what I found is that with my view with code review was always that code review is this kind of proxy for quality you know and we did it because it worked really well in the human world. It was a really effective way of humans finding bugs. But in the AI world there are a lot of other things and other doors that suddenly open. You know I we we write pretty much exclusively in Rust. A little bit of TypeScript for front ends but mostly Rust and um for that you know suddenly you've got mutation testing and you got property testing you got fuzz testing. You've got a whole range of things which in the past even the doors opening to MCDC multiconditional um uh decision um checkpointing testing which is the kind of stuff you would use in the past if you were building an aeroplane you know control system you know it's really expensive you're checking all the possible paths through the code suddenly the doors start opening to all this other world of testing and coverage that you can do and you can still do code review but the agents just do code review every night for me um and they find a lot of stuff out that way it turns comes out um and then you know the bugs drain automatically. It it's all a bit kind of weirdy science fiction. I find it really hard but you know at the end of the day you get results and and I I you know it was a bit like when compilers came along in the in in the early 80s. I'm not quite old enough to remember that but you were writing a lot of code in assembler and then there was this magic of this thing called compiler and people stopped looking at the assembly code and it took a while for that to happen and it feels like that's a sort of an analogy for what's sort of happening now that actually you gain the trust and you don't really need to look at the code um and then when you see the results that all the code works you know you sort of think well that's that's sort of job done isn't it um I that that that's where I am at. >> I have so many questions, but my first one is and I welcome I think Erin also has many questions. It's fascinating and you are I I do think you are well based >> over on the other side of the the world in the UK. You are in what David would call the Silicon Valley state of like this more extreme, but you are a twoerson, two human uh team right now. >> Okay. Do you enjoy it? >> Yeah. I think >> as someone who managed humans for most of their life life, engineering humans, >> I I I I miss that bit. I miss having a team. I particularly I always like managing new grads. Um, you know, it was always brilliant having new new people join the team and you could influence them and and hopefully you did things that were good and set them on a good good path for the rest of the career. I always I really really enjoyed that. Um, but I always like building things um and for me creating the product and getting into people's hands and seeing people using it and seeing them be excited by that that was for me much more important than actually, you know, the act of writing code. Code was a means to an end. Um, >> so interesting to me because every engineer I've ever known, including the one I'm marrying, they went into this this career because they they enjoy like solving a puzzle. And it's not just the getting to the end goal, it's the act of of solving that puzzle. So, I'm I'm quite curious that um for for folks who have that sort of mindset of this is how I work and I've got this big gory gritty thing to figure out, especially in my world with a big massive incumbent legacy system that we've got to deal with. Like, where does that then come from if what we're doing is giving code instructions um and letting it run? It almost feels like you're losing a big chunk of of what drew people to that career. Especially, again, I love managing grads. I love watching them figure out how to use their brains in this way, no longer in a classroom environment. It's great. Um, so what does that mean for them, I guess? >> Sorry, I was going to say I think it's I think that's a really interesting point. I think I think you still get a challenge of of of problems to solve. Um, you know, one of the things, one of the things the emulator can do is it can do 3D graphics, you know, and I thought it'd be really interesting to see what the wireframes are like without the textures on them. I wonder what that looks like, you know, and that's like five minutes. It's not five minutes, it's like half an hour's work. I wonder if I can use AI to upscale the textures that we're feeding in and into the graphics engine and have that working in the background. Well, you know, I can do that as well. So there are all these interesting problems I think where you think wonder how how that would work and play out and and it's like you've got this playground now. So it's not quite the same >> but it's >> I still find it really intellectually stimulating. >> Yeah, I agree. >> You think of these things. >> I agree with that. Yeah, I think I think even even when when answers are cheap, knowing the right questions to ask becomes extremely important. And I think knowing what you want to build, what problem you're trying to solve, and what good or done looks like also becomes sort of more the critical point. So we still get to choose to solve a problem in a specific way because ultimately I still have to feed the instructions in and what I choose to put in or leave out, the decisions I make along the way, what tests make it valid versus ones that make it invalid. those decisions like if I take two people who are going to build the same piece of software and I give them the agent systems that are going to write all the code and none of them are going to look at it the choices that they make are still going to lead to two different pieces of software like the the thing that the product at the end is going to look different is going to feel different it's going to potentially have features that are important for one person are going to be in one and not in the other and vice versa and so I think there's still a lot of creativity >> and a lot of um judgment and taste that comes in into play that's going to dictate what comes out at the end like you're talking about the textures and whether I want to look at it. That's a choice because of your own interest and things that that like spark something in you and someone else will find something different. Right. >> Okay. How do you figure out what's good if you've never coded before? If you're a baby grad, baby as in new, not human child. >> How do you know what is what looks good if you've never written that? So >> whether it's code or systems or project, >> you can think of it, but think of it like how do you know when you look at a painting whether you like that painting or not? It it has an impression on you, right? So I'm making a decision based on my tastes. Now you're saying like what looks what's good is going to be like well am I trying to sell it? So then I need not just care about what I think is good. It has to be something that other people think is good as well. And so if I happen to align in what I think is good with a lot of people, then probably my software will be successful and a lot of people will buy it. If I if someone else thinks something is good and only they really like it, it's not going to go very far. And I think you're right, they don't necessarily know, but you have a ton of new grads coming out creating companies using AI as as a as a lever to really move quickly and bring something to market. And their taste, their their perspective at the world and the problems they're trying to solve is having an impact, right? They're able to build things and then create companies that do very well without having been taught how to write good software. They have opinions. They have these desires to solve problems. >> I don't disagree with you. I think Gen Z are going to save the planet. Um I'm kind of obsessed with them. Um but I think what what is interesting to me and I know we've like Martin and I have briefly mentioned this and talked about this is if we're talking about maybe a few people with an idea that they're going to go and fix. But if I'm coming at this from like I've worked in in the banking industry and some of these banks are really old. So So as much as I would love to be able to pick up something that um works within its bubble, there's well there's two parts to this. There's one like if it works in its bubble, will it work really nicely with the rest of everything going on? And how do we make sure that that happens? But then on the flip side of it, the all of that, the stuff that you and Martin really get to do because you're in such a like a smaller, more nimble, more creative way of working, we just cannot do. Like we have so many restrictions. So what does that then leave for >> the engineers and the teams that I'm with >> who are told use AI, become more efficient, but frankly, they're going to spend more of their time undoing what Claude has written because it doesn't work with our our stack. So that it's it's there's almost like it's like a multiclass system in the tech world of who can actually get the best advantage out of some of these tools. Um which feels quite unfair sometimes. >> Yeah, green field green green field is definitely a little bit easier right now. >> Yeah. >> Legacy systems. That's for sure. >> I I wonder if it plays out like the you know the innovators dilemma. You know, you look at the hard disk manufacturers, you know, through the 60s and the 70s and 80s are the classic example here where as hard disks, you know, got smaller and smaller. The people who were the incumbents in the previous generation never managed to transition successfully to the next generation, new companies appeared and stole their market. And and you know, there is a bit of me that sort of thinks is there a risk that that plays out right now? you know, because someone like Microsoft for example, you know, you've got the office office and Word and Excel and things and you know, it's a good old cash cow, but they're ripe for reimplementation at this point. I don't know many people who would say they loved Excel or Word. Um, and um, you know, if >> jobs rely on it, the people that are going to lose their that's been always the problem, specifically the US government. There are people whose whole jobs in life have been Excel spreadsheets and if Excel gets automated, their job is done. And that is a valid concern, but not the most thrilling one. It's possibly the most flippant thing I'm gonna Well, I'm gonna say it's the most flippant thing I'm gonna say tonight and I will contradict myself later down the line, but guaranteed like the instant someone automates something in an indust like in a place like that in government or in like a third sector, there is one woman called Brenda who is keeping everything running with her secret Excel that is still going on because what what the AI agent that's been shoved into the company doesn't know is all of the weird like unwritten processes and who you have to talk to that's secretly about that and then it becomes more of like a feeding something from a secret spreadsheet and she is the only person in the office that knows what's happening. There's always one. >> Yeah. Okay. So, we've got a very Thank you. Thank you to the audience. We've got a lot of questions. So, we move on from this. Multiple people have asked you Martin is this just spec driven development? What instruments, materials, documents define your spec? Is there a common set of those? Are they dynamic for project dependent? And we also have another question that uh yeah, it sounds like you're leaning into spec driven development and maybe test-driven development, behavior driven development to validate the output of coding agents. So somebody's reviewing your agents then, right? So, I'm going to throw this to Martin first because there are a lot of questions like, "Oh, but what are you doing?" Because I think a lot of people want to, but no one's near it. >> I I I I like the term Oracle driven development, not just because the acronym is quite odd. Um, but uh yeah, so or or you start off with your oracle. How are you going to test this this thing? And you build those first, you know, and and um you know, you can have a whole variety of different oracles which and you know, the idea of an oracle is that is this imperfect source of truth that tells you whether it's you know, you're implementing. So I implemented a SIP stack. So you know the SIP standards are one example and then there's a whole lot of other examples of SIP stacks which you can use which are they're not all perfect and they cover in different places, you know, so you but you can take the union of them and get get more coverage. Um and then once you've got that then it's very much it is it's it's spec and designs and it's just a classic software um engineering process where I'm very heavily involved in what the requirements are for what we're we're trying to achieve and what the spec and as the you know as we get further down the design I hand off more and more trust to the agents and I'm less and less involved and then they're implementing that and and there's multiple agents in this and there's adverse adversarial reviews you know and you've got called workflows coming together with multiple agents looking at things with with different viewpoints. Um, and then you leave them to it and you've got your oracles at the end and you know you could have you know the the the emulator as a performance oracle. Um, you know, and Claude was Claude Fable I think worked quite happily with Opus for I think about five days on that you know to to drive the performance of this to where where I wanted it to be. Um so you can get I think if you get that sort of framework around your agents they can just run for a very long time and and they are um very um tenacious you know they will they won't stop. >> Okay. >> Does that give you a sort of a bit of a flavor for >> it is a flavor. It is I think it brings up a lot more questions than answers from the audience which I love. So that's great. Um and as we said like questions are more important than ever if people have learned to ask the right questions I think is the thing. So Theo asks how should teams verify that coding agents follow architectural decisions which I would add a little subparenthesis. How many companies are actually writing and maintaining architectural decision records or anything like that? End of parenthesis to continue the thoughts. uh not how many so how should teams verify that coding agents follow architectured decisions not just that the generated code passed tests which we know testing's something that follows a pattern just like documentation it's something that AI is very good at especially in doing things that we were never good at or exclusive and then do you see deterministic guard rails before or during generation complementing postgeneration review So AI is very non-deterministic. So what guard rails allow you to move fast and can they >> Well, it's interesting. So um so the question about deterministic guardrails and the other question was about um >> architectural decisions >> architectural decisions. So um for the deterministic guardrails, it's interesting. One of the things I think with light writing in Rust is Rust basically gives you a lot of guard rails for free. Um there's a there's a lot of stuff you know you get and a whole class of bugs that you know you can't have scribblers effectively I mean you can use onslave rust and things and you get yourself into trouble if you want to but but generally you know and concurrency problems so the things that I spent you know weeks of my life on debugging in when I was working in C they've gone this is nice um in terms of the the flow everything that the agents do is journal they all I mean I used to some of these are things I taught the graduates when they came. You know, I I was always a fan of having a paper and pen and a log book, you know, and you wrote down what you did and that was your journal. So, that's what my agents do. And and that turns out to be immensely valuable because it does two things. One, you know, it's another pass through the LLM. So, that when they're thinking about something, they they writing the act of them writing it down makes them think, "Oh, I've missed a thing." Right? And and that sometimes they catch things that way. And secondly, it's just there's a record that you can go back and find out what happened, which you have in addition to the session logs, but session logs are per machine. Um, which you know, if you've got a fleet of machines, that's a that's a problem. And that's always going to be a problem where you've got multiple um engineers. Um I I think as well what you were touching on at the beginning about knowing the right question to ask in David's point. I think that's an incredibly important point as well there of you know it's it's I find a lot of it is there's a thing I had as a manager of you know people would come and they would talk to you about something and and and all of it would feel fine but every so often your sort of spidey sense would go off and you think I don't understand that that just doesn't smell right that there's something wrong there and you dig into that and and you I do the same thing with the AIS of that you know I don't understand that doesn't make any sense why why are we doing this do we need to do this? You know, that's another good question, you know, and that's as true, I think, with AI agents as it is with the humans. I I I think I'm quite inherently lazy and I I I don't like doing things unless I really have to. Um, and I used to do this a lot at work. You know, people would say, could you do this? And I say, why? Um, and you do the same thing with the agency. Why are you doing that? And and you know, eventually it'll collapse and they'll, yeah, you're right. We don't need to do that. You know, and and a piece of work disappears. Um, so I think I think actually you do knowing the right questions to ask and having I mean I'm really lucky. I've got all these years of experience and seen a lot of things and I think that's that turns out to be you know quite useful at this point. I think those that those years of experience though are are what I feel like gets gets you the end result that you want. And I think that that is my concern as a as a non-technical human and also my concern working with a bunch of very technical humans at the early stage of their career is how how do I get to that point? And um I know David you said earlier about it's knowing whether like you you sort of have the smell for when something is good and you know when to ask the right questions like if we remove it from this world entirely. I can go to a hospital and ask exactly the right questions and I have no idea if the answer I'm getting back is necessarily correct because I don't have medical experience. >> For sure. >> And and it's kind of it's it's that sort of thing. And I I think I think what is possibly missing in some some places and it's something that we're talking about in my current place is where does the buck stop and who is realistically accountable if and when something goes wrong. I think in Martin your situation it's a lot more obvious there's only two of you that kind of is where it goes. >> Yeah. But I I don't know if any of you saw the the story recently where you know Amazon are now using AI agents in their chat function and a customer very angrily demanded a phone number to talk to a person and ended up in a very notfor line and there is literally no one for them to blame until you hit the chief execs who have never really interacted with the chat bots. So it's understanding like how to build it into the timeline that a person with experience can actually verify what is coming out of that and also that they have the experience to be able to do that. >> But Martins in the chat asked if engineers stop reviewing code should future engineers know how to read or write code is that just not part of the job anymore so review becomes moot or soon enough? No, I don't think review becomes moot because there's still a point of understanding even if you're not looking at the code. So if I if I go back to some things that Martin said like he notices things and he gets them to change direction. So if you don't look at what's happening even at a higher level, then you're never going to catch those things, right? >> And there's still some things that a agents will do that are nonsensical to people who have more engineering background like the infrastructure they'll choose to use or or things like that won't make sense. The scale of your company or various other choices just don't make sense. And that has nothing to do with the functionality necessarily of the product. It can just be a decision like do we use Kubernetes versus using cloud functions or some other thing that's simpler earlier on. But I I I think the review is also a knowledge sharing thing. if I want to know. So for example, last weekend we were building two systems separately and then we wanted to join some of the base together and have it reuse some some common elements and it the AI system worked for I think around 40 hours and it did it but it also built in by itself. It chose to build in another level tier to the second function. So we had a basic tier and a deep tier and it put in a middle tier and we didn't ask it to do that. Now if I didn't look at the if I didn't look at the output from the high level and I just get a summary of what's going on what did it do then I would never catch that right I don't need to look at the code to catch it I can I can look at what happened and code rabbit provides this for example as one of our functions like you can look at what happened at a high level using natural language and it will explain to you exactly what went on so you can break the problem down and understand even in the case where I I do believe that we're going to move similar to compilers so we'll we won't necessarily look at the code and rust is a great example like Rust is a great language for AI coding. There's so many protections in that particular case. So many things that are deterministic that will prevent the AI from just going off and doing things that that will break systems. But you still at this point, if you're going to deploy something, you're going to need to know how does this thing go out? How do all these systems connect? Is this thing correct for me? I don't want a $10,000 Amazon bill in a couple of days because it decided to to create some system for millions of users when I have none. So like there there's there's different things that we still need to do, but I don't think it necessarily means we need to look at the code. >> Yeah, thank you Ariton in the chat who said, but this is a knowledge of an experienced person. >> Yeah, >> is a compliment. How are we going to have people leaving uni getting that experience? It's not looking at something and seeing if it feels good. And then we also have Martin's asked, "If we have so many agents reviewing and building critical parts of our code, aren't we creating a layer of unpredictable inflation, which is money?" And then this is I asked the group because I think Erin's question about accountability is very interesting because you should be in charge of whatever is releasing, but who's in charge? So, and I asked the the chat like, "Is accountability clear in your companies when it comes to AIdriven deployment development?" And Michael said, "We haven't had an issue yet where accountability counted, but I'm worried about it when it does." So, how just as we're winding down because we are lead dev, how does the manager's role change when the manager is like Martin said a second or third line manager? Managing people, managing agents. So, is managing agents different? A lot of these people have never managed anyone before in the lowest level. And then from there, how do we manage people managing AI? What changes in the review like of everything? How are we going to make sure if we're not doing humanto human review? >> Well, we've we have put things into into our sort of working agreements with some of our teams like nothing goes to a live environment that has not passed through human eyes. And that that has been our first step. And then it really does become the the person who signs it off. Now again, I'm going to point out I'm working in a legacy company. I I'm an insurance right now. Literally nothing goes like automated at all ever like regardless of who wrote it. So it kind of works for our SDLC that that we can sort of build that in and say it gets to a point where yeah it may have been automatedly automatically written but a person has to look at it and that is all part of our audit and that person becomes accountable for making sure that it's not just functional but it's also secure because we've got to think about cyber security and and all of those weaknesses as well as I'm sure like everyone on this this call does. So that that's been our first step. But again that works because we have a massive department and our like our percentage of AI gen code like purely gen code versus augmented is very different to possibly what some of y'all are doing. So I'm very curious actually what the answer is to this in the long run. >> Yeah. >> I think I'm curious. >> I don't know that anyone's found the exact right answer. Right. I think we're we are dealing wrestling with a brand new technology that has the capacity and it's constantly changing to be fair but that to do a lot right and the question is what are the guardrails fit who does someone have to look at from my perspective my engineers the expectation is judgment rests in the hands of the engineers >> and at the end of the day they have to understand they don't have to read every line of code but they have to understand the things that went in and that's where like our system is meant and we we use our own system extensively to help people understand and point to if there is a risky thing like we will flag it. This is a high-risk piece of code you should check this out. This is modifying a database schema. This is modifying external facing APIs. You're modifying something and we have this notion of a blast radius architectural reviews. Did you radically change the architecture of the system? There's a lot of things that we will try and help the reviewer do a better job at focusing attention because PRs are getting larger and larger in our in our case because we're also like Martin writing a lot of our code >> using agents like our CEO even writes code has five agents running in the background almost at all times building stuff right and so at the end of the day he's taking responsibility for the things I'm taking responsibilityility for the things I'm putting in it it stop it stops with the individual still at our company even though people are very very heavily using AI. >> I do think it's true that nobody has figured it out. But somewhere along the line, we've stopped sharing what's gone wrong. And as a journalist, I beg you to tell those stories because first of all, I think that's probably the best advertising for recruitment your company could ever have is this is how this messed up and how we fixed it because >> yeah, things mess up all the time. >> Super important. But no, I mean to the general public, not just Code Rabbit. I meant like everyone when things happen like Capital One's the only one that's told me on the record, >> wow, we released this AI and a year later we were across the board we we got feedback they didn't like it, we got rid of it. >> And that to me was good advertising for the company. Uh we've got some Arton shared a policy at their office which it's around this which I think is very good to share. Own the output. AI is a helper not a replacement. You are fully responsible for the accuracy and quality of your work. >> AI made it. You own it. Always fact check and review AI suggestions before you use them. >> And I thought that was excellent. and Margarita who says I don't see agents as individuals not yet like code they should be owned and maintained by a team that takes responsibility for them and that's a good point are we breaking down the whole system of teams so it's individual developers reviewing everything I keep hearing one of the most troubling thing I've heard is I pair program with AI and that's >> yeah I mean we don't do that in our I consider ourselves to be very AI pill we don't we don't go so far is to that that you get to choose whether something is is done or not. You definitely should know exactly what went in and you should be the first like gate between if something is going to be looked at by somebody else. And then ultimately at the end of the day, every company's going to be different. But if you're at this stage where you're saying, "Okay, nothing can go in without me having understood the code lines of code." I think that that's going to allow someone else to come in who doesn't have that and come at it from a different perspective. Like if you if you're doing the things that Martin's doing, right, you're going to be very fast moving, you're going to fix issues as they come in very rapidly. And so as long as you're adding guardrails and you're adding more and more things to make sure that the blast of anything bad happening, it keeps getting mitigated, you can move quite fast. I I still think understanding like buck stops with the person who put it in. But I do think that there's not a need in the over the next little bit of actually knowing every line of code that's going into your system. >> And I I I the analogy I use is it's a bit like when I first became a manager and when I was first a manager, it was it was really scary because I lost, you know, I didn't have control over everything. I didn't know all the code that was going in. I had to trust other people. I couldn't review every line of code. There was too much going in. You know, once you have a team of six people, you you can't review everything. So you have to trust people and you end up with proxies and things for you know and figuring out where the dangerous bits are and the bits that you you you trust. And I think it's very much the sort of same analogy with AI that you're you know yes I'm still responsible for it but I know the bits that I'm going to go and and be suspicious and question and the bits that yeah I that's fine. I'm going to I'm going to trust you. M >> it's kind of building that trust I think with it which is a weird thing to be building trust with a machine but there you go. it that that is a phrase that sort of makes me a little bit what inside because I I think I'm permanently conscious of the fact that and I think I said this to you before as well like I if we look at who's created these tools um how do I know as as someone who is not a straight white dude that if I was to use aentic AI to make something I'm not going to be negatively um impacted and like we're seeing studies come out on like the sort of things that that might have enduser impact purely because no one no one cognitively thought about it and went through AI systems. And I think that's the other thing that people need to think about, not just the the fine bone detail, is when we're looking at stuff that goes out into the big bad world, are we making sure that it's fit for the current world and the world that we're moving to. I think that this becomes a really big like esoteric philosophical question that I have no answer to, but I like to make people think about um >> because it's permeating into every single corner of my life. And I don't know if you can tell, I quite like things in the physical format. >> I'm sure if Martin turned off or David turned off or if I was at home, we'd all be in the same physical book plate. uh we are definitely going to continue the chat in the lead dev slack. So I welcome people both uh that are joining now or watching the recording later to join us there. And I just thought uh as a last comment Louie in the audience brought up as part of a small team something we do is challenge the owner of a pull request if we suspect they don't have a high level understanding of the solution. Maybe don't say it that way though. uh they should be able to explain what the design is, what overall decisions were made even if they don't know every line. That allows us to maintain an understanding of the system without getting too bogged down in the syntax. And I would say to everyone using AI, take a moment to reflect on that. Even if you're not, if you're doing it by yourself in a personal project, be beholden at least to yourself to reflect those decisions and why you're making them. Speed for speed has always led to crap. Uh it's not going and users are going to reach a point when they have too many features very soon. So let's do that. I will see you all on the slack and thank you so much Martin, David, and Aaron. This was a great conversation and I think it's the very beginning of a very long discussion and that is a good thing. We don't have all the answers. No one does. And please keep sharing your questions and your experiences. Thank you everyone. Good luck y'all. >> Thanks for having me. >> Thanks for having us. Thank, Effective Verification for Humans and Agents, YouTube, 01kzxxnca3c1vape5j6w0zmvj6.