Lean TDD With Brian Okken  🔗 **Source:** [Lean TDD With Brian Okken](https://www.youtube.com/watch?v=2Y7fOH3VleU) > [!note]+ Document Notes > tags: [lean-tdd, test-driven-development, system-level-testing, test-suite-waste, ai-coding-agents, pytest] > author: Brian Okken, Brittany Ellich & Erika Eggemeyer > source: Overcommitted (YouTube) > reference: https://www.youtube.com/watch?v=2Y7fOH3VleU > date created: 2026-09-23 10:25 > > > [!info] Generated analysis — not the author's words. Sections below are machine-written > > from the captured source. Only text inside `> [!quote]` blocks is verbatim from the author. > > Do not quote or attribute any other line to the named author. > > # 🏷️ Title: Lean TDD with Brian Okken > <!-- deep-read:sections start --> ## 🧭 Table of contents - [📝 Summary](#-summary) - [📖 Main content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning styles assessment](#-learning-styles-assessment) - [📊 Accessibility assessment](#-accessibility-assessment) - [🌍 DEIB assessment](#-deib-assessment) - [✅ Logical validity](#-logical-validity) - [🛡️ DARVO analysis](#️-darvo-analysis) - [🎯 Conclusion](#-conclusion) - [🔄 Refactor appendix](#-refactor-appendix--lean-tdd-with-brian-okken) ## 📝 Summary - **What happened.** The Overcommitted podcast (hosts Erika Eggemeyer and Brittany Ellich, who met on a team at GitHub) published a 41:41 interview on 2026-09-22 with Brian Okken, lead engineer at Rohde & Schwarz, author of *Python Testing with pytest*, host of the Test & Code podcast, and author of the self-published 100-page book *Lean TDD*. The auto-transcript renders his name as "Brian Aen," his employer as "road and Schwarz," the hosts as "Erica" and "Britney," pytest as "pi test," and the book's site as "pythonest.com." Quotes below keep the transcript spelling. - **The claim.** Write each test at the highest level that still makes sense for the job, so that refactors, restructures, hardware swaps, and agent-driven edits don't break the suite. Function-level tests with heavy mocking are harder to write than system-level tests because nobody knows the requirements of an interior function. The "TDD takes too long" complaint is a symptom of testing at the wrong level. The rule that tests must be fast dates from the 1990s, when developers ran everything on their own machine; CI changed that. And a test suite carries waste the same way source code does, so tests written on the way to a higher-level test get deleted once the higher test passes. - **Why it matters.** Coding agents refactor cheaply and add tests eagerly. A suite tied to function signatures turns every agent refactor into a test-rewrite, and an agent told to "do TDD" can produce more test churn than code churn (a host reports this at 00:20:58). A suite anchored to the system's public API survives both. - **Key stakeholders.** Developers using pytest, PHPUnit, Playwright, or any test runner; teams adopting coding agents; test engineers whose roles carry a pay-scale stigma (00:26:49); Kent Beck, whose "Canon TDD" post Okken cites as endorsing wider-than-unit scope (00:10:45); the lean software development lineage Okken borrows the word "lean" from (00:24:38). - **Risks and tradeoffs.** System-level tests localize a failure less precisely than a unit test does, and the episode doesn't address that cost. Okken concedes hardware-backed suites stay slow (00:23:10). His career evidence is one person's account (00:07:57 to 00:09:21). No number in the episode is measured: "5 minutes" is a tolerance, "20 vs. 1,000 tests" is a thought experiment, and "stuff is fast now" is an assertion from a 2019 MacBook Pro. Okken is selling the book he describes, and says so. - **Bottom line.** The mechanism holds without the book: a test that fails when behavior is unchanged carries no shipping information, and the higher the API a test targets, the less often behavior-preserving change reaches it. The rest of the episode is practice built on that one rule. ## 📖 Main content **The core argument.** Okken's version of test-driven development starts from what a developer already does: try the thing out. Every programmer who ships software checks it somehow, by clicking, by calling it from a REPL (which the transcript renders as "ripple"), or by writing a throwaway wrapper. TDD, in his telling, is writing that wrapper down as a test function so a runner can repeat it (00:02:54 to 00:03:37). The cost over manual checking is small; the benefit is that the check keeps running. The "lean" modifier answers a specific failure. Teams refactor constantly, and with agents they refactor more, because "move this over there" or "split this up" is now a one-line request (00:04:19). If the tests target individual functions, every such refactor breaks tests that were never signaling a defect. The system's API, by contrast, rarely changes; when it does, that's a deliberate behavior change or a major version (00:04:19 to 00:05:01, 00:14:22). So: put tests as high as possible, and refactor underneath them. **Why function-level tests are harder, not easier.** The standard teaching says isolate the unit, mock everything around it, and test in place. Okken says he never learned to do that, and that even if he could, he wouldn't know what the function's requirements are. An interior routing function that moves a packet from a button click to hardware has no independent spec; its valid inputs are whatever the system feeds it. Run the whole system and you can observe that the function only ever receives the numbers 1 through 7, so the negative-number test you'd have written in isolation is waste (00:06:30 to 00:07:13). This is the episode's strongest technical point: the requirements needed to write a unit test often don't exist until you've observed the system. **Terms defined.** - *Red-green-refactor*: change the test (it fails, red), change the code (it passes, green), then tidy. Okken keeps the rule that you don't change test and code at the same time, and notes that mock-heavy TDD forces you to break it (00:13:38 to 00:14:22). - *Unit test*: Okken's history is that "unit" named the framework family in the 1990s (unit test frameworks), and the "smallest possible thing" meaning was attached later by others (00:10:45). He credits Kent Beck's Canon TDD post (transcript: "Kat Beck," "Canon TD") with saying the scope is wider than functions. - *Acceptance criteria*: plain statements of what the software should do, such as "a calculator should add two numbers." Okken dislikes the formality of the term and has no better one (00:34:41). - *Acceptance test driven development (ATDD)*: writing those criteria as automated tests first. His company is adopting a version of it and the book exists partly so he can hand it to colleagues instead of retelling the backstory (00:35:24). - *Waste* (lean): time, code, or tests that don't contribute. A test that's outdated or redundant costs someone a fix-or-delete decision later (00:24:38, host's framing). Okken took "lean" from lean manufacturing via a book he calls "lean software development," whose first part covers principles and whose later two-thirds gives practices he found didn't all transfer (00:24:38 to 00:25:20). He doesn't name the authors; the title and structure match Mary and Tom Poppendieck's 2003 book. **The speed rule is a period artifact.** "Big tests are slow, unit tests are fast" made sense when the suite ran on the developer's own machine and a slow suite meant nobody ran it. Okken runs component tests locally on a 2019 MacBook Pro, pushes, and lets CI run the whole suite; if CI takes five minutes he gets a notification and fixes it after his coffee (00:23:10 to 00:23:53). He excepts hardware-backed suites, which stay slow regardless. **Deleting tests.** A Django developer who read an early draft described his habit: start small with tests around one area, widen to the database, then write Playwright (transcript: "playright") tests through the UI. The realization was that once the UI-level test passes, the earlier tests are disposable, because if a small one fails while the large one passes he won't act on it (00:21:40 to 00:22:25). Okken extends this to agent-written tests: delete what isn't helping. His one standing rule for agents is to put each test at the highest level that makes sense for the job (00:22:25). **Agents and reading code.** Okken treats agent output like an intern's: he reads every line because he has to maintain it, and he doubts the "next year's model will maintain it" excuse (00:18:47 to 00:19:30). He hasn't tested Simon Willison's (transcript: "Simon Willis") suggestion of telling the agent to do red-green-refactor, but reports that when a good suite already exists, agents augment it usefully, sometimes with tests he deletes because they embed customer data (00:20:13). A host adds the counter-experience: harnesses that bake in TDD add so many overlapping or contradictory tests that she refactors tests more than code (00:20:58). **Test count as a vanity metric.** If someone implements in 20 lines what took you 100, you're impressed. If someone has 1,000 tests to your 20, you're also impressed, and Okken asks why, since you wouldn't admire 20x larger source (00:26:05). **Roles and career.** Okken describes a stigma against test engineers reinforced by pay scales, argues the role needs a software engineer's skill plus communication and tolerance for vagueness, and proposes a division that's safe for both roles: developers build the skeleton of the system and the skeleton of the tests, then test engineers extend into corner cases, timing, security, and localization (00:26:49 to 00:29:40). His personal claim is that caring about the whole system rather than his slice got him noticed and moved his career (00:07:57 to 00:09:21). He warns against wanting an autopilot job, because those are the jobs going away (00:30:23). **The book itself.** Self-published because Pragmatic (his pytest publisher, rendered "Pragmat") wanted length and examples he didn't want to add; 100 pages, sized for a flight. He recorded the audiobook himself and didn't release the text until he'd read every paragraph aloud, cutting lines that sounded pompous and softening snark at other testing schools (00:32:34 to 00:33:59). Distribution is Amazon and Audible only until a 90-day exclusivity ends in November 2026 (00:41:09). **Uncertainties and contested claims.** Nothing in the episode is measured. The claim that system-level tests are cheaper over a codebase's life is a mechanism argument, not a study. The failure-localization cost of system tests goes unmentioned. The career story is one case. The "most people teach it at the function level" claim names no curriculum. Kent Beck's endorsement is Okken's summary of a post, not a quote. ## 🗒️ Notes #### ⏱️ [00:00:00](https://www.youtube.com/watch?v=2Y7fOH3VleU&t=0) Intro, hosts, and what lean TDD is - Hosts introduce themselves as Erica and Britney (Erika Eggemeyer and Brittany Ellich per the description); they met on a GitHub team. Guest introduced as lead engineer at Rohde & Schwarz, creator of the pytest book and course, author of *Lean TDD*. - Icebreaker: Okken wants to build a SaaS for estate sales with texting, outside his embedded-systems background, and hopes an agent closes the web-development gap (00:00:43). - For someone who's "never written a test," Okken doesn't believe them: every developer checks their work manually or via a REPL; TDD writes that check down as a function the runner calls (00:02:10 to 00:03:37). - The lean part: teams refactor constantly, agents make it cheaper, and function-level tests break on every refactor while the system API rarely changes (00:04:19). > [!quote] [00:05:01] > the idea around lean TDD is just to say let's try to get the tests as high as possible so that we can refactor without breaking tests. #### ⏱️ [00:05:01](https://www.youtube.com/watch?v=2Y7fOH3VleU&t=301) "TDD takes too long" is a level problem - Okken agrees it takes longer "if you do it the way most people teach it," at the function level (00:05:45). - Worked example: a tiny sort utility. You'd probably use an off-the-shelf sort; if you wrote one, nobody will change it, so the test rarely earns its keep (00:05:45 to 00:06:30). - Interior code has no independent requirements. Observing the system shows the function only receives 1 through 7, so isolated edge-case tests are waste (00:07:13). > [!quote] [00:06:30] > I think it's harder to write tests for small things than for the system because what is it supposed to do #### ⏱️ [00:07:57](https://www.youtube.com/watch?v=2Y7fOH3VleU&t=477) Caring about the system, and what it did for his career - Asking "how do I drive it from the outside" meant his tests never needed rewriting because the API never changed (00:07:57). - Side effects: learned the system, understood customers, ran demos at customer sites, found other teams' bugs and got them fixed by direct message in minutes instead of via triage (00:08:39). - He says the visibility moved his raises and his career (00:09:21). One person's account; no comparison group. - He discourages "cover yourself" testing while admitting the impulse is real; the company has to make money for anyone to get a raise (00:09:21 to 00:10:04). A host adds that reorgs make "not my problem" temporary (00:10:04). > [!quote] [00:07:57] > All of that time I gained back because I didn't have to rewrite my tests because the API never changed. #### ⏱️ [00:10:45](https://www.youtube.com/watch?v=2Y7fOH3VleU&t=645) Where "unit" came from, Canon TDD, 1990s GUI automation - Okken's history: "unit" named the framework family; the "smallest possible thing" definition was attached later by someone else (00:10:45). - Kent Beck's Canon TDD (transcript "Kat Beck," "Canon TD"), published roughly a decade ago by Okken's estimate, describes a scope wider than functions (00:10:45 to 00:11:28). Verify the date; the post is from 2023. - 1990s instrument GUIs on Windows and Sun Solaris were automated by pixel coordinates and clicks; companies hired teams of button-pushers instead (00:11:28 to 00:12:12). > [!quote] [00:10:45] > it was called units and unit testing because the test frameworks were called unit test frameworks for some reason. #### ⏱️ [00:12:56](https://www.youtube.com/watch?v=2Y7fOH3VleU&t=776) The workflow: toggle test and code, and let stable APIs enable rewrites - Host observes the book's workflow never goes on autopilot: it questions names and user clarity at every step, and separates test edits from production edits (00:12:56 to 00:13:38). - Okken: red-green-refactor already says don't change both at once; mock-heavy TDD forces you to violate it because the test encodes the structure (00:13:38 to 00:14:22). - Higher in the API, the less that happens; customer-facing APIs don't get their parameters rearranged outside a major version (00:14:22). - Payoff: rewrite an algorithm you don't understand, run the suite, `git revert` if it fails (00:15:06). - Kent Beck's framing (from an interview Okken recalls): imagine all permutations of characters in a codebase; tests should reject the broken ones and keep the ones you'd be proud of (00:15:49 to 00:16:33). - With a language-agnostic API, such as REST or the instrument API in Okken's embedded work, the same tests survive moving software to hardware, hardware to an FPGA (transcript "FPJ. J"), or Python to Node, on codebases that are decades old (00:16:33 to 00:17:18). > [!quote] [00:15:49] > If if the test can break just because I refactored something but the functionality of the system's still great that's not a good indicator. #### ⏱️ [00:17:18](https://www.youtube.com/watch?v=2Y7fOH3VleU&t=1038) What tests are for: "does it work," not taste or performance - Host: automated tests get used for things they weren't designed for. CI query-counter tests that fail when a change adds database queries have good intent but don't answer "does it work" (00:17:18 to 00:18:02). - Host: unit tests grew up alongside pair programming and carry a "software taste" function, and you can't write a unit test for taste (00:18:47). #### ⏱️ [00:18:47](https://www.youtube.com/watch?v=2Y7fOH3VleU&t=1127) AI-written code, reading every line, agents and tests - Okken notes people are now proud of code they can't read because an AI wrote it, and he thinks that's a bad idea (00:18:47). - He doubts "next year's model will maintain last year's model's code" and reads everything, then adds that if you don't, you'll want tests even more (00:19:30). - He wanted to write this book for 20 years; agents made the audience ready, because people not reading low-level code won't write unit tests for it (00:19:30 to 00:20:13). - Simon Willison (transcript "Simon Willis") proposes telling the agent to do red-green-refactor; Okken hasn't tested that thoroughly. His experience: with a good suite in place, agents add useful tests, and he deletes the ones that leak customer data (00:20:13). - Code style is fine; group ownership doesn't require uniformity, and you can ask an agent to describe a function (00:20:58). - Host: prebaked TDD harnesses add too many tests, some duplicating or contradicting each other (00:20:58). > [!quote] [00:18:47] > I treat the AI stuff like an intern or a new hire > [!quote] [00:20:58] > I find myself having to like refactor the tests almost more than the actual code that is being changed #### ⏱️ [00:21:40](https://www.youtube.com/watch?v=2Y7fOH3VleU&t=1300) Deleting tests, the one agent rule, and the speed myth - A Django developer's progression: tests around a small area, then the database layer, then Playwright through the UI. Reading the draft made him realize the earlier layers can go once the UI test passes, because a failing small test with a passing big test won't change his action (00:21:40 to 00:22:25). - Same applies to agent-written tests, which Okken suspects exist partly to bill tokens (00:22:25). - Okken has few rules for agents; the one he names is the level rule. - The "big tests slow, unit tests fast" rule is from the 1990s. He runs a full local stack on a 2019 MacBook Pro without thinking the suite is slow, except for hardware-backed tests (00:23:10). - The old rule assumed developers ran suites locally and would skip slow ones. Now component tests run locally, the merge request triggers the full suite, and a five-minute CI run is acceptable (00:23:53). - Host ties deletion to lean waste: outdated code in the repo costs someone a fix-or-delete decision later (00:24:38). > [!quote] [00:22:25] > one of the rules is put the unit test or the put the test at the highest level that makes sense for this job. > [!quote] [00:23:53] > and if it runs for 5 minutes I don't care. #### ⏱️ [00:24:38](https://www.youtube.com/watch?v=2Y7fOH3VleU&t=1478) Lean principles, and why 1,000 tests shouldn't impress you - "Lean" comes from lean manufacturing via a book Okken calls "lean software development"; the principles transferred, some of the practices didn't because the authors wrote different kinds of software (00:24:38 to 00:25:20). He wants to introduce more developers to lean thinking. - The asymmetry: 20 lines beating 100 impresses; 1,000 tests beating 20 also impresses, but 20x more source wouldn't (00:26:05). - Host: line count oversimplifies; 10 hard-to-read lines waste a future engineer's time versus 20 clear ones (00:26:05 to 00:26:49). > [!quote] [00:26:05] > So why why am I impressed with larger numbers of tests? #### ⏱️ [00:26:49](https://www.youtube.com/watch?v=2Y7fOH3VleU&t=1609) Test engineer stigma, the skeleton design, and autopilot - Okken's experience: test engineers were treated as less advanced and paid less, though the role needs engineering skill plus communication and tolerance for vagueness (00:26:49 to 00:27:32). - Two frictions: software engineers refusing system tests because "it tests other people's code," and test engineers fearing for job security if developers write system tests. His answer: there's more to test than anyone can cover (00:27:32 to 00:28:14). - Developers must be able to read and debug system tests when CI fails, without tossing them to a test engineer for log-gathering (00:28:14). - Skeleton design from the book: build the skeleton of the system and the skeleton of the tests together, hand off, and let test engineers extend into corner cases, timing, security, localization, and CI maintenance (00:29:40). - Warning against autopilot jobs (00:30:23). > [!quote] [00:27:32] > This doesn't seem like less skilled than a software engineer. It seems like more. > [!quote] [00:30:23] > Don't don't wish that your job was autopilot because all the autopilot jobs are going to go away. #### ⏱️ [00:30:23](https://www.youtube.com/watch?v=2Y7fOH3VleU&t=1823) Shared ownership, meeting waste, and career advice - Host: "doesn't break other areas" and "no performance regression" belong in acceptance criteria even when you can't automate them; the on-call engineer, not the test engineer, gets paged (00:30:23 to 00:31:51). - Okken: caring about tests, CI, and the whole codebase compounds; lean applies to meetings too, and cutting 30 minutes to 25 returns 10 minutes to everyone's day (00:31:51). - If caring about the whole team doesn't help your career, "get another job because it should" (00:31:51). #### ⏱️ [00:32:34](https://www.youtube.com/watch?v=2Y7fOH3VleU&t=1954) Self-publishing a 100-page book and recording the audiobook - Self-published because Pragmatic (his pytest publisher) pushes for length and examples; he wanted something readable on one flight (00:32:34). - Recorded the audiobook himself, first time. Held release until he'd read every paragraph aloud, cutting pompous lines and softening snark toward other testing schools (00:33:17). - Advice: read your technical writing aloud even without releasing audio; more tech books should be small and available as audiobooks (00:33:59). - Host asks whether reading acceptance criteria or test descriptions aloud would help; Okken wishes "acceptance criteria" weren't so formal a term and would have used a better one if he had it (00:34:41). - His company is working on acceptance test driven development; the book saves him retelling the backstory (00:35:24). > [!quote] [00:33:17] > Wow, that just sounds pompous and idiotic. I can't I can't say that out loud. #### ⏱️ [00:35:24](https://www.youtube.com/watch?v=2Y7fOH3VleU&t=2124) Fun segment, HP barbecues, standups, and where to find Brian - Prompt: if you could be wasteful about one part of the software life cycle, what would it be? Erika: in-person offsites and a nice team dinner once a month or quarter (00:36:08). Brittany: planning and bikeshedding designs, plus 3D-printing system diagrams (00:36:53). Both recall crocheting GitHub commit-history blankets; the site name is garbled in the transcript (00:37:34). - Okken: at HP in Santa Rosa in the 1990s, a good quarter meant a same-day barbecue announcement, cheap relative to revenue but memorable (00:38:16 to 00:38:59). He ran a team for about 12 years and gave half of every standup to personal catch-up, which he can't measure but considers important for remote teams (00:39:42). - Find him at pythontest.com (transcript "pythonest.com"), which links to courses and leantdd.com. The book is Amazon and Audible only until the 90-day exclusivity ends in November (00:40:25 to 00:41:09). - Outro: follow the podcast, find them on Bluesky (00:41:09). ## 📚 References The transcript itself contains no links; the following come from the video description or are primary sources for named entities. - [Lean TDD with Brian Okken, Overcommitted](https://www.youtube.com/watch?v=2Y7fOH3VleU): the source video, 41:41, uploaded 2026-09-22. - [Lean TDD](https://leantdd.com/): the book's site, named at 00:40:25. - [Lean TDD, free sample chapter 1 (PDF)](https://leantdd.com/downloads/lean-tdd-chapter-1-sample.pdf): from the description. - [pythontest.com](https://pythontest.com/): Okken's blog, named at 00:40:25 as the hub for everything else. - [pytest course](https://pythontest.com/courses): from the description. - [Test & Code podcast](https://testandcode.com): from the description. - [Brian Okken on GitHub](https://github.com/okken) and [LinkedIn](https://www.linkedin.com/in/okken/): from the description. - [Overcommitted](https://overcommitted.dev), [Brittany Ellich](https://brittanyellich.com), [Erika Eggemeyer on GitHub](https://github.com/eggyhead): hosts, from the description. - [Canon TDD, Kent Beck](https://tidyfirst.substack.com/p/canon-tdd): the post Okken paraphrases at 00:10:45 as describing a scope wider than functions. Published December 2023, not "10 years or so" ago as he estimates. - *Lean Software Development: An Agile Toolkit*, Mary and Tom Poppendieck (2003): Okken names the title but not the authors at 00:24:38. The principles-then-practices structure he describes matches this book; confirm before citing. - [Simon Willison's blog](https://simonwillison.net): source of the "tell the agent to do red-green-refactor" suggestion Okken mentions at 00:20:13. Transcript renders the name "Simon Willis." - [pytest documentation](https://docs.pytest.org): the framework Okken's earlier book and course cover. - [Playwright](https://playwright.dev): the UI test tool in the Django developer's progression at 00:21:40; transcript "playright." - [Rohde & Schwarz](https://www.rohde-schwarz.com): Okken's employer; transcript "road and Schwarz." - [The Pragmatic Bookshelf](https://pragprog.com): publisher of *Python Testing with pytest*; transcript "Pragmatic" and "Pragmat." ## 🗣️ Paraphrase Okken's pitch is that you already test; you just do it by hand and then throw the evidence away. Write that check down as a function and a runner will repeat it forever. The only design decision that matters is where to aim it. Aim at a function and you've bound the test to the shape of the code, which means every rename, split, or move breaks a test that was never telling you about a bug. Aim at the system's API, the one customers or other teams call, and the test survives every refactor underneath it, because that API changes only when you mean to change behavior. The rest follows. The reason unit tests feel expensive is that you're inventing requirements for a function nobody specified; run the system and you find out the function only ever sees seven values. The reason people still insist on fast tests is a 1990s constraint (the suite ran on your laptop) that CI dissolved. The reason a thousand tests looks better than twenty is a bias you wouldn't apply to source code. And the reason to delete the tests you wrote on the way up is that a small test failing while the big one passes changes nothing about what you'd ship. Agents sharpen all of this: they refactor for free and add tests for free, so a suite anchored to structure turns into churn, and a suite anchored to behavior turns into an advantage you didn't pay for. ## 📰 Newsletter **Test where the promise is, not where the code is.** Brian Okken, the pytest book author, spent 41 minutes on the Overcommitted podcast making one argument: put every test at the highest level that still makes sense, so you can refactor, restructure, or hand a change to an agent without rewriting the suite. For WordPress plugin authors that lands on a familiar fault line. A PHPUnit test that mocks `get_option` and asserts on a private method's return is bound to the shape of your code. A test that installs the plugin in wp-env, hits the REST route or runs the WP-CLI command, and checks the rendered output is bound to what you promised users. Only one of those survives the refactor an agent will happily do for you at 2 a.m. Three things worth stealing. First, the test-count check: you wouldn't be impressed by 20x more source code, so why are you impressed by 20x more tests? Second, the deletion rule: once the Playwright test through the UI passes, the smaller tests you wrote on the way there are candidates for the bin, because a small failure with a passing big test won't change what you ship. Third, the speed myth: "unit tests must be fast" assumed the suite ran on your laptop; if CI takes five minutes, Okken doesn't care, and neither should you. The book is 100 pages, self-published, and Okken recorded the audiobook himself, cutting every paragraph that sounded pompous when read aloud. That's a technique, not an anecdote. Links: [the episode](https://www.youtube.com/watch?v=2Y7fOH3VleU), [leantdd.com](https://leantdd.com/), [free chapter 1](https://leantdd.com/downloads/lean-tdd-chapter-1-sample.pdf), [Kent Beck's Canon TDD](https://tidyfirst.substack.com/p/canon-tdd). ## ✅ Action items #### DevRel - Developer advocacy: build a 10-minute talk demo that refactors a plugin's internals with an agent twice, once against a WP_Mock-style unit suite and once against a wp-env PHPUnit plus Playwright suite, and count broken tests in each run. - Education: add a "which level does this test belong at" exercise to any testing workshop, using Okken's sort-utility example (00:05:45) and the 1-through-7 observation (00:07:13) as the two poles. - Developer experience: ship plugin scaffolds with a REST-level or CLI-level integration test as the first test, not a mocked unit test, so the default example models the level rule. - Developer marketing: no numbers to promote here. The episode's claims are mechanisms, not measurements; cite the mechanism. - Community management: the test-engineer stigma discussion (00:26:49) is a discussion prompt for contributor channels; frame it as a career topic, not a complaint. - Internal/external/cross-project: share the single agent rule from 00:22:25 with any team writing CLAUDE.md or AGENTS.md files; it's one line and testable. #### OSPO - Policy and governance: add a contribution-policy line that agent-generated tests are reviewed for embedded customer or personal data before merge (Okken deletes these at 00:20:13). - License compliance: no direct action from this source; nearest relevance is that agent-written test fixtures can carry copied data of unknown provenance. - Security management: Okken names security testing as a test-engineer extension layer (00:28:58). Treat security checks as system-level tests owned by someone, not as unit tests nobody runs. - Software inventory: no direct action from this source; nearest relevance is tracking which repos have a system-level suite at all. - Developer support: publish a one-page "read every line" expectation for agent output, matching Okken's intern framing (00:18:47). - Community engagement: no direct action from this source. - Strategic alignment: replace any "number of tests" KPI with "behaviors covered at the public API," per 00:26:05. #### Community - Maintainers: run a test-suite waste review: list tests that failed in the last quarter without a corresponding shipped fix, and delete or raise their level. - Contributors: give first-issue contributors a system-level test to extend (a corner case, a locale, a timing bound) rather than a unit test to write; Okken's skeleton model (00:29:40) makes that the natural handoff. - Sponsors: no direct action from this source; nearest relevance is funding CI minutes so the five-minute full suite (00:23:53) is affordable. - Ecosystem: no direct action from this source. #### WordPress - Core: Core's PHPUnit suite mixes function-level tests with integration tests against a live database. Audit which tests broke in the last three releases without a behavior change; those are structure tests by Okken's definition (00:15:49) and candidates to move up to the REST or e2e layer. - Core Performance: the host's query-counter example (00:18:02) is Core Performance's territory. Keep performance assertions in a separate suite with explicit budgets, not mixed into "does it work" tests. - Core AI: if agents are used on Core, the level rule at 00:22:25 belongs in the guidance for them, and agent-added tests need a deletion pass (00:20:58). - **Core Program Team:** the skeleton-then-extend handoff (00:29:40) is a coordination pattern between feature developers and the test team; write it into the release retrospective template as a question. - Mobile: no direct action from this source; nearest relevance is the same level rule applied to app UI tests. - Playground: Playground is the fast, throwaway system-level environment Okken's speed argument (00:23:10) assumes exists. Document a blueprint that boots a plugin plus fixture content so a PHPUnit or Playwright system test can run against it in CI, and measure the wall-clock to test the five-minute claim. - Community: no direct action from this source. - Training: a Learn WordPress lesson on "choosing the level of a test" with the three-layer Django progression (00:21:40) translated to PHPUnit, wp-env, Playwright. - Marketing: no direct action from this source. - Support: no direct action from this source; nearest relevance is that system-level tests catch the bugs users report and unit tests often don't. - Design: no direct action from this source. - Documentation: the Plugin Handbook's testing pages lead with unit tests. Add a section that puts REST, CLI, and rendered-output tests first and mocking last, with Okken's reasoning at 00:06:30. - Photos: no direct action from this source. - TV: no direct action from this source; nearest relevance is caption accuracy, which this episode's auto-captions show is needed for proper nouns. - Polyglots: Okken names "all the languages of the world" as a corner-case layer for test engineers (00:28:58). Locale coverage is a system-test extension, not a unit-test concern. - Accessibility: no direct action from this source; accessibility checks sit at the same system level as the Playwright layer. - Meta: no direct action from this source. - CLI: WP-CLI commands are a language-agnostic API in exactly Okken's sense (00:16:33). Test plugins through `wp` commands where they expose them; those tests survive PHP refactors and would survive a rewrite. Add smoke tests at the command level to the CLI scaffolds. - Hosting: no direct action from this source; nearest relevance is CI runners fast enough that a full suite finishes in minutes. - Tide: no direct action from this source. - Openverse: no direct action from this source. - Themes: no direct action from this source; nearest relevance is that theme tests are inherently rendered-output tests, already at the right level. - Plugins: for plugin authors with PHPUnit and Playwright suites, the concrete change is the deletion pass: for each Playwright flow that passes, list the PHPUnit tests covering the same behavior and remove the ones that only assert on internal structure. Then add the one-line agent rule (00:22:25) to the repo's agent instructions so new tests land at the right level. #### The WPOCC - Programs: no direct action from this source; nearest relevance for The WP Open Community Collective is that a maintainer stipend could fund a test-suite waste review on a widely used plugin. - Projects: no direct action from this source. - Funding experiments: if a microgrant funds test coverage, require the deliverable be stated as behaviors covered at the public API, not test count (00:26:05). ### FAIR - Technical independence: a suite written against a plugin's public contract (hooks, REST, CLI) is what makes an independent reimplementation verifiable; Okken's Python-to-Node point at 00:16:33 is the general case. - Policies and covenants: no direct action from this source. - Productized governance artifacts: a scorecard row for "has a system-level test suite that survives refactors" is measurable and cheap to check. ## 🧪 Hypotheses 1. In a WordPress plugin with both PHPUnit and Playwright suites, a behavior-preserving refactor (renaming and moving internal functions with no hook, REST, or CLI contract change) breaks a higher fraction of PHPUnit tests than Playwright tests, and the gap exceeds 30 percentage points. 2. When an agent-driven refactor breaks only function-level tests, the agent edits the tests rather than the code in more than half of cases, producing the test churn a host describes at 00:20:58. 3. Deleting lower-level tests whose behaviors are covered by a passing system-level test reduces a plugin's mutation-testing kill rate by less than 5 percentage points. 4. A full wp-env PHPUnit suite for a mid-size plugin (200 to 500 tests) completes in under five minutes on a 2019-class laptop, so the speed premise for isolating unit tests doesn't hold for that class of project. 5. Giving an agent the single rule "put the test at the highest level that makes sense for this job" yields a suite with fewer tests and equal or higher public-API behavior coverage than giving it "use TDD." 6. Across public WordPress plugins with CI, test count correlates weakly (Pearson r below 0.3) with the rate of user-reported regressions per release. ## 📊 Methodology - **H1.** Pick five plugins with both suites. Apply a scripted refactor (rector or a rename-and-move script) that preserves behavior. Run both suites; record failing tests per suite over total tests per suite. Confirm behavior preservation by a manual smoke pass so failures aren't real regressions. - **H2.** Take the H1 refactors and hand each to two agents with the instruction "make the tests pass." Diff the result: count commits that touched test files versus source files, and classify each test edit as "loosened the assertion," "updated to new structure," or "fixed a real regression." - **H3.** On the same plugins, run a mutation tool (Infection for PHP) before and after deleting PHPUnit tests whose asserted behavior is exercised by a passing Playwright flow. Report kill rate and the list of surviving mutants introduced by deletion. - **H4.** Time `wp-env start` plus `phpunit` cold and warm on a 2019 MacBook Pro or equivalent, three runs each, for five plugins. Report p50 wall-clock and the share of time spent in environment boot versus tests. - **H5.** Two agent runs per feature across 10 features: one with Okken's level rule in the repo instructions, one with "use TDD." Count tests written, and map each test to a public-API behavior; coverage is distinct behaviors hit, not lines. - **H6.** Sample 100 plugins on GitHub with CI badges. Count tests from the CI log and regressions from issues labeled bug that reference a release within 30 days. Compute correlation and report the confidence interval; expect noise, and say so. - **Controls throughout.** Pre-register thresholds. Fix agent model and version across arms. Publish the refactor scripts so runs are reproducible. ## ❓ Questions - What's the failure-localization cost of a system-level suite, and at what suite size does debugging a red system test cost more than maintaining the unit tests it replaced? The episode never addresses this. - Where does "highest level that still makes sense" stop for a WordPress plugin: the hook contract, the REST route, the WP-CLI command, or the rendered page in Playwright? - Okken's 1-through-7 observation (00:07:13) came from watching real data flow. How do you get that observability in a plugin that runs on sites you don't operate? - Does the deletion rule (00:22:25) hold under mutation testing, or do the deleted small tests catch mutants the big test misses? - Kent Beck's Canon TDD post is from December 2023, not roughly a decade ago as Okken estimates (00:10:45). Does the post say what he says it says about scope? - What did the Poppendieck practices that "didn't apply" look like, and which lean principles did Okken keep in the book? - The host's query-counter test (00:18:02) fails Okken's "does it work" filter. Where do performance budgets live if not in the test suite? - Okken hasn't tested Willison's "tell the agent to do TDD" approach (00:20:13). Has anyone published a comparison of agent-written suites under the two instructions? ## 🔮 Future research - **Level selection as a discipline.** There's no published heuristic for choosing a test's level beyond "as high as makes sense." A decision table keyed on API stability, failure-localization cost, and setup time would make the rule teachable. - **Agent test churn.** The host's report at 00:20:58 that harnesses add more test edits than code edits is an observation nobody has measured. Instrumenting agent sessions for test-file versus source-file edit ratio is cheap. - **Deletion safety.** Whether scaffolding tests can be deleted without losing mutation coverage is directly testable and hasn't been, as far as this episode shows. - **CI economics of system tests.** Okken's five-minute tolerance (00:23:53) is a personal threshold. The cost curve of system-level suites in CI minutes, against the developer time saved on refactors, would settle the speed argument with numbers. - **Test engineer role design.** The skeleton-then-extend division (00:29:40) is a proposal. Case studies of teams that adopted it, with before and after defect and retention data, don't exist in this source. - **Reading aloud as review.** Okken's audiobook process (00:33:17) is a review technique for prose. Whether reading acceptance criteria or test names aloud catches ambiguity is an open, cheap experiment. ## 💡 Implications - **Practice.** The default first test in a plugin changes from a mocked unit test to a REST, CLI, or rendered-output test. Agent instructions get one line: put the test at the highest level that makes sense. Test suites get a scheduled deletion pass. - **Policy.** Team standards that mandate unit tests per function, or set test-count targets, encode the bias Okken names at 00:26:05. Replace them with behavior-coverage statements at the public API. - **Security.** Okken places security testing in the test engineer's extension layer (00:28:58). That's a system-level concern, and it needs an owner, because unit tests that mock the request never see the injection. Agent-written fixtures also need a data-leak review before merge (00:20:13). - **Education.** Curricula that teach TDD as function-level red-green-refactor produce the "takes too long" reaction at 00:05:45. Teaching level selection first, and mocking last, changes the first experience. - **Funding models.** CI minutes become the budget line that makes system-level testing viable. Grants and stipends that fund "test coverage" should specify the level, or they'll fund the count. - **Careers.** Okken's claim that caring about the whole system moved his career (00:09:21) and his warning about autopilot jobs (00:30:23) are the same argument: thinking about the system is the non-automatable part of the job. ## 🧠 Learning styles assessment - **Reading level estimate.** Flesch-Kincaid Grade Level on the transcript body: approximately grade 7, with Flesch Reading Ease near 75. Conversational sentence length and common vocabulary keep it low; the domain terms (mocking, stubbing, red-green-refactor, REST, FPGA, CI) assume a working developer. #### 👀 Visual learners - Present: nothing. This is a talking-heads podcast with no slides, diagrams, or code on screen. - Improve: a single diagram of test levels (function, component, system API, UI) with a refactor arrow showing which levels it crosses would carry the whole argument. The three-layer Django progression at 00:21:40 is a natural second figure. #### 🔊 Aural learners - Present: the entire episode is conversation, and Okken's own audiobook practice (00:33:17) is the topic at one point. Music stings at open and close only. - Improve: chapter markers by topic; a two-minute recap of the level rule and the deletion rule at the end. #### 🗣️ Verbal learners - Present: Okken defines TDD from first principles (00:02:10 to 00:03:37), gives the sort-utility example, and offers the 20-lines-versus-1,000-tests analogy (00:26:05). - Improve: "mocking," "stubbing," "acceptance criteria," and "FPGA" go undefined in the moment; one clause each would help newcomers the intro claims to address. #### 🧩 Kinesthetic learners - Present: the workflow description at 00:12:56 to 00:14:22 and the "rewrite it and revert if it fails" habit at 00:15:06 are actionable. - Improve: no repository, no sample suite, no exercise. The free sample chapter in the description is the closest thing; a companion repo with the Django three-layer example would let listeners do it. #### 🧠 Logical learners - Present: the mechanism argument (refactors change structure, tests bound to structure break, the API rarely changes) is laid out in order at 00:04:19 to 00:05:01, and the deletion rule at 00:22:25 has a clear decision logic. - Improve: the failure-localization tradeoff is never stated, so the argument reads as one-sided. Name the cost and the logical learner can weigh it. #### 🧑‍🤝‍🧑 Social learners - Present: the hosts push back with their own experiences (query counters at 00:18:02, agent test churn at 00:20:58), and the test-engineer stigma discussion (00:26:49) invites reaction. - Improve: no community link beyond "find us on Bluesky." A discussion question per episode would help. #### 🧘‍♀️ Solitary learners - Present: the book is designed for a single flight (00:32:34), and the episode is self-contained. - Improve: a reflection prompt such as "list the tests in your suite that failed last quarter without a shipped fix" would turn the episode into an exercise. #### 🌿 Naturalistic learners - Present: lean's manufacturing roots (00:24:38) and the waste framing are systems thinking, if not ecological. - Improve: CI minutes have an energy cost; a suite that runs fewer, higher tests uses less compute. Naming that would land with this group and is consistent with the lean argument. ## 📊 Accessibility assessment #### Screen reader semantics - Present: YouTube supplies page landmarks and player controls. The description includes plain-text links to every resource named aloud. - Improve: no on-screen text to describe, so 1.1.1 (Non-text Content) has little to fail. The garbled proper nouns in the auto-transcript are the main semantic gap; see Media and captions. #### Keyboard navigation - Present: the YouTube player is keyboard operable with visible focus by default. - Improve: nothing in the content adds barriers. 2.1.1 (Keyboard) and 2.4.7 (Focus Visible) pass via the platform. #### Contrast and readability - Present: not assessable from the transcript; a podcast video typically shows a static or webcam frame. - Improve: if any lower-third text or title cards exist, they need 4.5:1 per 1.4.3 (Contrast Minimum). Unverified here. #### Text resizing and reflow - Present: player and captions scale independently. - Improve: burned-in title text, if any, doesn't reflow (1.4.4 Resize Text, 1.4.10 Reflow). Low impact for a talking-heads format. #### Media and captions - Present: auto-generated captions exist, which is how this analysis was produced. - Improve: the captions mangle the guest's name ("Brian Aen"), employer ("road and Schwarz"), both hosts, Kent Beck ("Kat Beck"), Simon Willison ("Simon Willis"), pytest ("pi test," "Piest"), Playwright ("playright"), FPGA ("FPJ. J"), REPL ("ripple"), and the blog URL ("pythonest.com"). That's a 1.2.2 (Captions, Prerecorded) failure in substance for every proper noun a listener would search for. Human-corrected captions for 41 minutes of two-to-three-speaker audio are a modest fix. No chapters exist, so topic navigation requires scrubbing; 2.4.10 (Section Headings) is AAA, but chapters are the podcast equivalent and cost minutes to add. Music overlaps the intro speech at 00:00:00 (transcript marks [music] mid-sentence twice); 1.4.7 (Low or No Background Audio) is AAA, so this is a quality note rather than an AA failure. #### Forms and interactions - Present: none in the video. Links live in the description as selectable text, which is the right place. - Improve: not applicable in-video. The book and blog URLs are spoken clearly at 00:40:25 and repeated in the description. #### Standards reference - Primary gaps against WCAG 2.2 AA: **1.2.2** (auto-captions inaccurate on every proper noun). Beyond AA but worth doing: chapters for navigation (2.4.10 AAA analog) and ducking the intro music under speech (1.4.7 AAA). No 1.2.5 (Audio Description) gap of consequence, since nothing visual carries information the audio doesn't. ## 🌍 DEIB assessment #### Cultural breadth - Present: examples come from US corporate settings (HP Santa Rosa, GitHub) and Okken's embedded-instruments domain. He does name localization ("all the languages of the world," name sizes) as a real testing layer at 00:28:58. - Improve: the test-engineer stigma is described as varying by industry and country (00:26:49) without examples from outside the US. One would widen the claim. #### Inclusive language - Present: gender-neutral throughout; the hosts and guest use "people" and "engineers." Jargon is mostly defined by example. - Improve: "weirdo" for Kent Beck (00:15:49) and "a little weird" for the Poppendieck book (00:24:38) are affectionate but read as ability-framing on paper. Okken himself describes cutting snark from the book after reading it aloud (00:33:17), which is the fix. #### Neurodiversity support - Present: the conversation stays on one thread at a time and Okken restates his thesis at 00:05:01, 00:15:49, and 00:22:25, which aids retention. - Improve: no chapters, no summary, and the fun segment at 00:35:24 shifts register without warning. Chapter markers and a one-line recap would help listeners who need structure. #### Representation - Present: two women hosts and a male guest; the hosts contribute substantive counter-experience rather than only prompting. Test engineers, a role Okken says is underpaid and stigmatized, are represented as skilled (00:27:32). - Improve: no test engineer speaks for themselves; the role is described by a software engineer. A follow-up with a test engineer guest would balance it. #### Psychological safety - Present: Okken states that people advocating other test designs "are just trying to make software better for everybody. They're not jerks" (00:33:17), and frames disagreement with the function-level school as a level choice, not a competence judgment. His "everyone is winging it" message in the description is aimed at newer engineers. - Improve: "if it doesn't [help your career], get another job" (00:31:51) is glib for listeners without that mobility. A qualifier would cost one sentence. ## ✅ Logical validity #### Strong reasoning - **Causal mechanism, 00:04:19 to 00:05:01.** Refactors change function boundaries; tests bound to function boundaries break; the system API changes only on deliberate behavior change; therefore tests bound to the API survive refactors. Each link is a claim about where change lands, and each is checkable in a repo's history. To clarify: state the exception (a refactor that changes the API is a behavior change and should break tests). - **Empirical scoping, 00:07:13.** Observing that an interior function only ever receives values 1 through 7 bounds the tests worth writing. This is inductive from observed data and it beats speculation about negative numbers. Clarify by noting the observation window: production traffic, not one dev session. - **Definitional argument, 00:15:49.** A test's job is to block shipping broken software; a test that fails when the software still works isn't doing that job. This is a valid inference from a stated definition of what tests are for. The definition itself is the thing to argue with, and Okken states it openly. - **Premise invalidation, 00:23:10 to 00:23:53.** The "tests must be fast" rule rested on the premise that developers run the whole suite locally and skip slow ones. CI removed the premise; the rule doesn't follow from nothing. Okken also states the surviving case (hardware-backed suites), which keeps the argument honest. - **Decision-theoretic deletion rule, 00:22:25.** If a small test fails while the higher test covering the same behavior passes, the developer's action is to delete the small test, not fix code. A test whose failure never changes an action carries no information. Valid, with the caveat below about mutation coverage. - **Analogy with a stated asymmetry, 00:26:05.** Nobody admires 20x more source code, so admiring 50x more tests needs a reason. This is an intuition pump, not a proof, but it correctly puts the burden on the person defending test count. #### Fallacies - **Anecdotal generalization, 00:07:57 to 00:09:21.** "It skyrocketed my career" is one person's account of one company, offered as a reason others should care about the system. *Fix:* keep the mechanism (visibility from cross-team bug reports) and drop the outcome claim, or present it as personal history. - **Unnamed opponent, 00:05:45.** "The way most people teach it" and "a lot of people teach that" (00:06:30) name no curriculum, book, or course. Okken partially repairs this by citing Kent Beck's Canon TDD as a counterexample (00:10:45), but the "most people" claim stays unsourced. *Fix:* name two widely used TDD tutorials that teach function-level isolation. - **Hasty generalization, 00:23:10.** "Stuff is fast now" from one 2019 MacBook Pro and one person's suites. He concedes hardware suites are slow, but doesn't address suites with large database fixtures or browser matrices. *Fix:* state the suite size and runtime he's describing. - **One-sided cost accounting, 00:15:06 to 00:17:18.** The benefits of high-level tests are enumerated; the cost, that a red system test takes longer to diagnose than a red unit test, is never mentioned. Not a formal fallacy, but an incomplete comparison. *Fix:* name the localization cost and say when it's worth paying. - **Date error, 00:10:45.** Canon TDD is estimated at "10 years or so" ago; the post is from December 2023. Minor, and it doesn't affect the argument. *Fix:* corrected in this note. - **Speculation flagged as such, 00:19:30.** "Next year's model might not understand last year's model's code I guess" is guesswork, and Okken labels it that way ("I don't know"). Not a fallacy; noted so it isn't quoted as a claim. - **Checked and not found:** no appeal to authority beyond citing Beck's post for a scope claim, no false dichotomy (Okken repeatedly allows lower-level tests where they make sense), no straw man of mocking beyond the unnamed-opponent issue above. #### Tone of voice - Conversational, self-deprecating, and direct. Okken undercuts his own pitch ("Now you don't have to buy my book, but there's more gems in there," 00:05:01), calls Kent Beck "a weirdo, but I like him" (00:15:49), and describes cutting his own pompous paragraphs (00:33:17). Contractions and second person throughout. The hosts match the register and push back with their own experience rather than deferring. The one hard edge is career advice at 00:30:23 and 00:31:51, delivered as warnings without qualifiers; everything else is offered as "in my experience." ## 🛡️ DARVO analysis Not applicable to this source. DARVO (Deny, Attack, Reverse Victim and Offender) describes a response to accusation, and this is a book interview with no accuser and no defensive posture. The closest thing to criticism is Okken's disagreement with function-level TDD teaching, which he frames as a level choice and softens explicitly at 00:33:17 by saying other test-design advocates "are just trying to make software better for everybody." For completeness across the three prongs: **Deny**: nothing is denied; limitations (hardware suites stay slow, he hasn't tested Willison's approach, his career story is his own) are volunteered. **Attack**: no critic or school is disparaged; the "weirdo" aside about Kent Beck is affectionate and paired with praise. **Reverse victim/offender**: no such move appears; the one grievance discussed (test-engineer pay stigma, 00:26:49) is raised on behalf of others, not himself. ## 🎯 Conclusion The durable idea is a definition and a consequence. A test exists to block shipping broken behavior; a test that fails when behavior is unchanged isn't doing that job (00:15:49). Behavior-preserving change lands on structure far more often than on the public API, so a test aimed at the API survives refactors that would shred a function-level suite (00:04:19 to 00:05:01). Everything else in the episode is practice built on that: interior functions have no requirements to test against until you observe the system (00:07:13), the speed rule was a 1990s constraint that CI removed (00:23:53), scaffolding tests get deleted once the higher test passes (00:22:25), and test count deserves the same suspicion as line count (00:26:05). What the episode doesn't give you is a number or a cost. No suite size, no runtime, no defect rate, and no mention of what a red system test costs to diagnose compared with a red unit test. The career evidence is one person's. Treat the argument as a mechanism to test in your own repos, not as a result. **Next steps for readers** maintaining WordPress plugins with PHPUnit and Playwright: pull the last quarter's CI failures and mark each one as "shipped a fix" or "edited the test." The second column is your structure-test inventory. For each Playwright flow that passes, list the PHPUnit tests asserting the same behavior and delete the ones that only check internal shape. Add one line to the repo's agent instructions: put the test at the highest level that makes sense for this job. Then time the full wp-env suite; if it's under five minutes, the speed argument against system tests doesn't apply to you. ## 🔄 Refactor Appendix — Lean TDD with Brian Okken > Density: 2–5 Literature atoms (H3 pad), 0–3 Permanent candidates (H4 pad). > Emitted: 5 Literature, 2 Permanent. Resonance filter applied: the career-outcome story, > the HP barbecue anecdote, and the self-publishing process were dropped as not reusable > across projects. > > Konik claim-header rules: present tense, declarative, ≤10 words, searchable, unique. > > Atomicity: one declarative sentence per atom. "and"/"also" → split. > > Tag pool: #bmf · #addendum · #xref · #articleseed · #storystem · #share · #aphorisms ### 📚 Literature Split Pad #### [H1] Tests sit as high as possible so refactors survive > [!quote] [00:05:01] > let's try to get the tests as high as possible so that we can refactor without breaking tests **Context & use**: - 00:05:01, the one-sentence statement of the book's thesis, right after the explanation that agents make refactors cheap and function-level tests make them expensive. - This is the rule everything else in the episode derives from. It's also the line to paste into agent instructions. **Tags**: #xref #aphorisms <!-- atomicity: 1 idea ✓ --> #### [H2] Small-scope tests lack requirements to test against > [!quote] [00:06:30] > I think it's harder to write tests for small things than for the system because what is it supposed to do **Context & use**: - 00:06:30, answering "TDD takes too long." Followed at 00:07:13 by the 1-through-7 observation: watching real data through the system bounds what an interior function needs to handle. - Inverts the usual claim that unit tests are the easy ones. The requirement gap, not the test mechanics, is the cost. **Tags**: #xref #articleseed <!-- atomicity: 1 idea ✓ --> #### [H3] Refactor-triggered failures are not shipping signals > [!quote] [00:15:49] > If if the test can break just because I refactored something but the functionality of the system's still great that's not a good indicator. **Context & use**: - 00:15:49, stated as the criterion for what a test in the suite must be: something whose failure blocks a ship. - Gives a mechanical audit: any test that failed without a shipped fix is a structure test by this definition. **Tags**: #xref <!-- atomicity: 1 idea ✓ --> #### [H4] Higher-level tests make lower scaffolding tests disposable > [!quote] [00:22:25] > So once I write a larger level test for something, I can just throw away the older one. **Context & use**: - 00:22:25, relaying a Django developer's realization from an early draft: small, database, then Playwright tests, and the first two go once the third passes. Extended in the same paragraph to agent-written tests. - The decision logic: a small failure with a passing large test doesn't change what you ship, so the small test carries no information. Check against mutation coverage before deleting wholesale. **Tags**: #xref #share <!-- atomicity: 1 idea ✓ --> #### [H5] The fast-tests rule is a 1990s artifact > [!quote] [00:23:10] > big tests are slow and unit tests are fast really like that's from like the '9s. Stuff is fast now. **Context & use**: - 00:23:10, followed at 00:23:53 by the premise the rule rested on: developers ran the whole suite locally and skipped slow ones. CI moved the full run off the laptop; a five-minute CI run is acceptable to him. - Hardware-backed suites are his stated exception. Time your own suite before applying this. **Tags**: #xref <!-- atomicity: 1 idea ✓ --> ### 🧠 Atomic Split Pad #### [P1] A test earns its place by blocking a ship **Claim** (one sentence, Courtney's voice): A test belongs in the suite only if its failure would stop me shipping, so a test that fails on a behavior-preserving refactor is testing structure and gets moved up or deleted. **Evidence pointer**: [[#H1]] · [[#H3]] **Scope & conditions**: Holds when the system has a public contract (hooks, REST routes, WP-CLI commands, rendered output) that changes less often than internals. Weakens when failure localization is the bottleneck, since a red system test takes longer to diagnose than a red unit test; the episode never prices that cost. Doesn't apply to tests that exist to pin an algorithm's exact output during a rewrite, which are deliberately temporary. **Links**: - [[Test at the public contract]] — because extends - [[Mocking in WordPress unit tests]] — because contrasts on where the test binds, structure versus behavior - [[Agent instructions for plugin repos]] — because applies-in the one-line level rule from 00:22:25 <!-- atomicity: 1 idea ✓ --> #### [P2] Test count is a waste metric, not a quality metric **Claim** (one sentence, Courtney's voice): I apply the same suspicion to a large test count that I apply to a large line count, because tests carry maintenance cost and a redundant one costs a fix-or-delete decision later. **Evidence pointer**: [[#H4]] · [[#H5]] **Scope & conditions**: Holds when a higher-level test covers the same behavior as the lower ones being counted. Doesn't hold for tests covering distinct behaviors, edge cases, or locales that no system test reaches; those add coverage, not count. Verify with mutation testing before a deletion pass, because the episode's deletion rule is argued, not measured. **Links**: - [[Lean waste in software]] — because extends - [[Coverage percentage as vanity metric]] — because contrasts on what's being counted, lines versus tests - [[Agent-generated test churn]] — because evidence-for the host's report at 00:20:58 that harnesses add more test edits than code edits <!-- atomicity: 1 idea ✓ --> ### ✏️ Fleeting Queue - [F1] Pull last quarter's CI failures across the plugin repos and tag each as "shipped a fix" or "edited the test"; the second set is the structure-test inventory per 00:15:49 #fleeting #inbox - [F2] Add "put the test at the highest level that makes sense for this job" (00:22:25) to CLAUDE.md in the plugin repos and watch whether the next agent PR lands tests at the REST or Playwright level #fleeting #inbox - [F3] Open question: for each passing Playwright flow, which PHPUnit tests assert the same behavior, and does Infection's kill rate drop if they're deleted? Tests H3 in the Methodology #fleeting #inbox - [F4] Time the full wp-env PHPUnit suite on the current laptop, cold and warm, to check Okken's five-minute tolerance (00:23:53) against reality #fleeting #inbox - [F5] Read Kent Beck's Canon TDD (December 2023) and confirm it says what Okken says about scope at 00:10:45 before citing it #fleeting #inbox - [F6] Article seed: "Test where the promise is, not where the code is," translating Okken's level rule to WordPress hooks, REST, WP-CLI, and rendered output #fleeting #inbox #articleseed - [F7] Try Okken's read-aloud pass (00:33:17) on the next set of Playwright test names and acceptance criteria; the host's question at 00:34:41 is whether it catches ambiguity #fleeting #inbox ### → Things 3 (max 2 actions) - [ ] Audit last quarter's CI failures in the plugin repos and list every test that failed without a shipped fix — test-suite waste review, 45 min #things - [ ] Add the one-line test-level rule from 00:22:25 to the plugin repos' agent instructions and download the free Lean TDD chapter 1 PDF — repo hygiene, 15 min #things <!-- deep-read:sections end --> 📄 **Full text:** [[Full Document Contents/Articles/Lean TDD With Brian Okken.md|Lean TDD With Brian Okken by Overcommitted]] ## Highlights - [00:04:19] that I can push up because we have GitHub and CI and like stuff where I want like I change some new code and I just want a something to run that checks everything to make sure I didn't break anything maybe before a merge or something. So we're trying to build up those automated tests to make sure that they work. But we do like to refactor and stuff and especially with the AI you can and agents you can say can we just like move this over there? This looks weird. Can we split this up and it'll just do that. But if you've changed a bunch of functions and you have tests at the function level it's it's going to break your test. So you got to rewrite the test. But we don't really usually change the API to the system that much. We have to say like I'm actually adding a new behavior. I'm changing a [00:05:01] definition of behavior. So the really the idea around lean TDD is just to say let's try to get the tests as high as possible so that we can refactor without breaking tests. That's pretty much it. Great. Now you don't have to buy my book, but there's more gems in there. So >> Oh, there is. Oh, there is. Starting with sort of picking up where the book starts of making the argument for TDD and why it's valuable. and you recognize that engineers who do know what TDD is often find reasons not to do it. One of which is that they say that it's going to take them more time to do TDD than the alternative. >> Yeah, it will if you do it the way most people teach it. So most people teach it ([View Highlight](https://read.readwise.io/read/01m37b1pvzbpm0xytt3cmak43y)) - Note: [H1] Tests sit as high as possible so refactors survive - Tags: atom/literature/h1 - [00:05:45] at that little function level thing and that's hard. So let's say I like I I'm I'm in the middle of like something massive and oh I got some data coming in. I just need to like you're not going to really write your own sort algorithm, but I'm just like like let's say I want to take all these these uh elements in and I want to sort them or something. I could [snorts] write like a test for my little sort utility function also some and but like you probably won't because you'll just pull off the shelf sort but okay in our theory that we don't have off-the-shelf sort. We're going to write a little maybe a test for that. But then I could just like try it out and and nobody's going to change the sort function so it's not going to break really is it? So do I really need the test? And so the decision of it's really [00:06:30] it's actually pretty hard. I think it's harder to write tests for small things than for the system because what is it supposed to do like and and I write so a lot of the code I write is in the middle like I'm I'm there's a big software in lots of teams there's there's lots of layers of stuff and I'm like maybe taking when somebody clicks a button a package thing comes down and I have to route it to the right hardware or something like that or I need to when a measurement's done grab some hardware data and like push it up to the UI. How do I test that? Do I really have to like stub and fake out and everything? And a lot of people teach that to say really I just want to test the function you're writing. So isolate everything else, fake it, mock it, all that sort of stuff. And I've never done that because ([View Highlight](https://read.readwise.io/read/01m37b1s5103k41v24n8zggzgs)) - Note: [H2] Small-scope tests lack requirements to test against - Tags: atom/literature/h2 - [00:15:49] breaks I'm not going to ship the code. If if the test can break just because I refactored something but the functionality of the system's still great that's not a good indicator. And that's one of the the uh there was an interview with I actually kind of like Kent Beck. He's he's a weirdo, but I like him. And one of the things he said in an interview was we want our tests to be like you can imagine all of the different permutations of like letters in a in a in some code. And some of them will be workable software and some of them will actually be workable software that you that you'd actually be proud of. And we want tests to be the ones that eliminate the bad stuff and just keep the good ones. And so there's a like and if we really if we had that if [00:16:33] we had an API that's so high that's not even language specific like a REST API or something [snorts] we could rewrite the entire thing in a different language. I could go from Python to to Node or something and ideally the test would wouldn't care about the implementation. Now I don't I and I actually often test at this high level usually not rest APIs but because I'm I'm in embedded systems but I the the API that we do call the instrument with is a language agnostic API and that even though we're not going to change languages we often take something that's in software and make it faster in hardware or take something in hardware but the hardware piece is like going out of date or our vendors freaking out so ([View Highlight](https://read.readwise.io/read/01m37b1w3q05rftb6k395cdr4w)) - Note: [H3] Refactor-triggered failures are not shipping signals - Tags: atom/literature/h3 - [00:21:40] and you know about how your software works from that perspective if nothing else than like not wanting to wait hours for your CI to finish. >> You can also throw away tests. That's fine if if you don't think that they're valuable. Like one of the common things I was talking with a Django developer and he said often he'll start like doesn't I don't know where to start. So, I'll start in a like a small area and I'll write tests around that and then I'll I'll go to like the larger area with the database and write tests to make sure that works and then I'll go through with like the user interface and then write some playright tests to make sure it works through that that that system and and when I was talking to him he was one of the early readers of the book and he said one of the things I never thought about is those those previous iterations of the tests I don't need to keep those around because if [00:22:25] they break I'm not going to fix anything that because the highle test still passes. is if the little one breaks, I'm just going to delete it. So once I write a larger level test for something, I can just throw away the older one. And we can do that with agent test, too. If if an agent feels like it wants to do that, or more likely, it really wants to have tests so that it can bill you for tokens, you can you can delete those afterwards if they're not helpful. But I I also I I actually have one I don't have very many rules with agents cuz I I don't know. seems like they should if they work right, they should just get what I say. But but one of the rules is put the unit test or the put the test at the highest level that makes sense for this job. And in the old thing like so a ([View Highlight](https://read.readwise.io/read/01m37b1z8nbvhaqzkzb5a5x4d8)) - Note: [H4] Higher-level tests make lower scaffolding tests disposable - Tags: atom/literature/h3 - [00:23:10] lot of the things of like well big tests are slow and unit tests are fast really like that's from like the '9s. Stuff is fast now. We got fast computers with lots of memory and I run a whole bunch of stuff off of a 2019 laptop. Um, and yeah, it was a MacBook Pro, but it's from 2019. And it's still I never like I never think, oh, the suite's too slow uh for small stuff for like for big stuff with hardware. Of course, it's going to be slow. But a lot of that's a that's another thing is you got to get around is a lot of the unit test driven develop the test-driven development people will often say your tests have to be super blazing fast because there was this idea that if they weren't fast people weren't [00:23:53] going to run them because they were running them on their computer. We don't run them all on our computer anymore. or I run I run like my components tests on my computer and then I push the code up and do a merge request and then some magical thing goes and like tests all of my code and uh and it runs the whole suite and if it runs for 5 minutes I don't care. I'll get a I'll get a ding in 5 minutes that says there's a code error and I'll like interrupt my coffee break and go fix it. What you just said about deleting tests makes a lot of sense with the lean mentality too where you talk about what waste is. Lean is the opposite of wasteful and you talk about waste means wasted time. Wasted code. Code that's outdated or not needed ([View Highlight](https://read.readwise.io/read/01m37b21kkgne6hhawc8h1hjw4)) - Note: [H5] The fast-tests rule is a 1990s artifact - Tags: atom/literature/h3