Harness - What Breaks When AI Opens Pull Requests

Harness - What Breaks When AI Opens Pull Requests ![rw-book-cover](https://d34adp677peecb.cloudfront.net/static/images/article1.be68295a7e40.png) 🔗 **Source:** [Harness - What Breaks When AI Opens Pull Requests](https://www.youtube.com/watch?v=9vwISRj3V9s) > [!note]+ Document Notes > tags: [gitops, ai-generated-pull-requests, code-review, platform-engineering, delivery-metrics] > author: LeadDev > source: LeadDev (YouTube) > reference: https://www.youtube.com/watch?v=9vwISRj3V9s > date created: 2026-10-20 13:53 > > > [!info] Generated analysis — not the author's words. Sections below are machine-written > > from the captured source. Only text inside `> [!quote]` blocks is verbatim from the author. > > Do not quote or attribute any other line to the named author. > > # 🏷️ Title: Harness - What breaks when AI opens pull requests >  ## 🧭 Table of contents - [📝 Summary](#-summary) - [📖 Main content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning styles assessment](#-learning-styles-assessment) - [📊 Accessibility assessment](#-accessibility-assessment) - [🌍 DEIB assessment](#-deib-assessment) - [✅ Logical validity](#-logical-validity) - [🛡️ DARVO analysis](#️-darvo-analysis) - [🎯 Conclusion](#-conclusion) - [🔄 Refactor appendix](#-refactor-appendix--harness---what-breaks-when-ai-opens-pull-requests) ## 📝 Summary - **What happened.** LeadDev published a 47:16 live panel (uploaded 2026-10-20) on GitOps in the era of AI-generated pull requests. A host moderates three panelists and two audience polls. The video description is empty and the video has no chapter list, so every name and fact below comes from the transcript. The transcript has no speaker labels, so attributions are inferred from self-introductions and context. - **The commercial relationship.** The host opens by saying the panel is a LeadDev discussion created in partnership with Harness (00:00:05). One of the three panelists, Martin Reynolds, introduces the role field CTO at Harness (00:04:33). The video title leads with Harness. The panel is therefore partner content, not an independent editorial discussion, and Martin Reynolds' recommendations (AI-assisted risk scoring, AI test generation, continuous artifact verification) sit in the product space of the sponsor. The host discloses the partnership once, at the start, and the other two panelists come from Zalando and a health-data company. - **The claim.** The panel splits on one question: does AI change GitOps? Larry Garfield says no, because a pull request gets the same automated tests and human review whoever or whatever wrote it (00:15:34). Martin Reynolds says yes, because AI helps at many points along the pipeline: risk assessment of pull requests, test generation, drift analysis, deployment verification (00:16:59 to 00:18:27). Shyamala, an engineering leader at Zalando, lands between them: the fundamentals hold, volume is higher, and deterministic work should stay deterministic (00:19:10). - **Why it matters.** The title promises what breaks when AI opens pull requests. The answer given is that little breaks if the existing gates are sound: testing, staged rollout, observability, revert-by-pull-request. What breaks first is the reviewer, whose workload grows while team size stays flat (00:23:33). Weak pipelines fail faster, because AI amplifies the setup it lands in (00:33:45). - **Key stakeholders.** LeadDev (publisher and host), Harness (named partner, employer of one panelist), Zalando (runs 250+ clusters, per the panelist), a health-data company the transcript spells two ways, engineering managers deciding whether to adopt GitOps, and reviewers absorbing AI pull-request volume. - **Risks and tradeoffs.** Most statistics come with no source: above 90% GitOps adoption (host, 00:00:52), 45% of LLM code with security vulnerabilities (Martin Reynolds, 00:30:08), a minus 5 hour exploit-to-publish gap (00:30:51), and 81%, 78% and 63% survey figures (00:39:42). The live polls report no sample size. One panelist's blanket opinions about how LLMs are built (00:15:34) and about AI writing tests (00:16:16) are stated as fact with no evidence. - **Bottom line.** The strongest content is operational: a four-stage rollout with layered alerting, one panelist's rule that deterministic work should not go to a model, the same-model-marks-its-own-homework rule for test generation, and a metrics answer built around lead time and trend. The weakest content is the numbers. Treat the statistics as claims to verify, and treat Harness-employee recommendations as informed but interested. ## 📖 Main content **The setup.** The host frames the session with two numbers the host says came from research and does not source: above 90% of enterprises have adopted GitOps, yet most apply it to under half their production apps and only about 16% follow four or more of the assessed practices (00:00:52 to 00:01:35). The first poll (00:05:14): just over half the audience says it is still learning what GitOps is, around 10% is evaluating or piloting, and about a third says it is standard practice. The second poll (00:09:39 to 00:10:24): about two-thirds say AI opens pull requests and a human reviews every one before merge, about 22% say some AI pull requests merge with light or no review, and the rest say there are no AI-generated pull requests in their GitOps repo. **Definition.** Martin Reynolds defines GitOps as using Git as the source of truth for desired state, with four practices: declarative, versioned, pulled automatically, and continuously reconciled. Martin Reynolds says reconciliation is the one most teams miss (00:06:41). Larry Garfield adds that the pieces stack: a team can do infrastructure as code without GitOps and still gain value, and each added piece adds benefit (00:07:26). **The disagreement over AI.** Shyamala describes Zalando's posture: many pull requests now come from coding agents or with agent help, the flow resembles Dependabot with higher volume, and four-eyes review, end-to-end tests of the Kubernetes platform, traffic switching, an LLM first-pass review and an LLM risk label all apply (00:11:53 to 00:12:37). Shyamala objects to the risk label: the model lacks context, so a low-risk label can be wrong, and people may trust it because a machine said so (00:13:21). Larry Garfield argues the opposite pole: AI authorship is orthogonal to GitOps, and oversized pull requests get sent back whoever wrote them (00:15:34 to 00:16:16). Martin Reynolds argues AI changes the pipeline wherever it can read at scale: pull-request risk assessment, test generation, and drift analysis across clusters (00:16:59 to 00:18:27). **Where the panel converges.** All three agree that deterministic work should stay deterministic and that models earn their cost on judgment calls: incident first reports, whether to do task A or B, pattern-spotting across deployments (00:19:10 to 00:22:08). All three agree the guardrails are old practices: automated tests, observability, rollback, staged environments (00:28:40 to 00:29:22). They differ on how much those practices need to change. Martin Reynolds says security scanning, SBOM freshness and per-deploy verification must run at machine speed (00:30:51 to 00:31:36). **Adoption barriers and metrics.** Barriers named: no clear benefit gap over existing CI/CD, estates that deploy to Windows VMs, ECS or Lambda instead of Kubernetes, fear of a complex system, and lack of trust in your own tests and reviewers (00:35:15 to 00:38:14). Metrics named: business outcomes first, then lead time from pushed pull request to production, DORA measures, and trend over snapshot (00:41:07 to 00:46:15). **Contested or unverifiable in this source.** The host's adoption statistics, the 45% vulnerability figure, the minus 5 hour figure, the survey percentages and the claim that GitOps maturity tracks DORA performance carry no method or citation in the transcript. Larry Garfield's statement that LLMs are built on copyright infringement, sweatshop labor and environmental destruction is an opinion presented as fact (00:15:34); the other panelists neither endorse nor rebut it. Martin Reynolds' remark that a model company with $40 million in seed funding is small by AI standards (00:21:24) is also unsourced, and it concerns Jev, a product unrelated to Harness. **Name and term corrections supported by the transcript itself.** The captions render GitOps as githops and GitHubs; context makes GitOps certain. Zando and Zelando are Zalando (the speaker names fashion e-commerce as a past domain and describes a tech-first retailer). Lead dev is LeadDev. Dora is DORA. Platform SH is Platform.sh, and Upson is most likely Upsun, the new name for that company; the transcript gives no spelling and the description is empty, so that one is a probable correction, not a confirmed one. Syntacter and syntax are two caption spellings of one employer name the transcript never resolves; this analysis leaves it unresolved. The caption spelling of the first name, Shyamala versus Shamala, is also unresolved; this analysis uses Shyamala, the spelling the host uses in the introduction. No surname is given for Shyamala. The phrase dora dex pace at 00:42:36 is probably DORA, DevEx and SPACE, but the transcript does not confirm it. ## 🗒️ Notes #### ⏱️ [00:00:05](https://www.youtube.com/watch?v=9vwISRj3V9s&t=5) Opening, partnership statement, first polls - The host sets a 45-minute format, then a Q&A in the LeadDev Slack. - Research claims offered as homework for the audience: GitOps adoption above 90%, under half of production apps covered, about 16% following four or more practices. - Two polls launch: where the audience sits on GitOps, and where it uses AI in the pipeline. > [!quote] [00:00:05] > Today's panel is a lead dev discussion created in partnership with harness. > [!quote] [00:01:35] > only about 16% follow four or more of the assessed practices #### ⏱️ [00:02:19](https://www.youtube.com/watch?v=9vwISRj3V9s&t=139) Panelist introductions - Shyamala: engineering leader, 17 years across fintech, automotive and fashion e-commerce, platform systems where reliability affects customers. The host later places this panelist at Zalando. - Larry Garfield: developer for about 20 years, former developer relations at Platform.sh (caption: Platform SH), now at an employer the captions spell syntacter. - Martin Reynolds: field CTO at Harness for almost three years, 30+ years building and delivering software. > [!quote] [00:03:48] > recently I joined syntacter > [!quote] [00:04:33] > field CTO at uh Harness #### ⏱️ [00:05:14](https://www.youtube.com/watch?v=9vwISRj3V9s&t=314) Poll 1 results and the definition of GitOps - Just over half of respondents are still learning what GitOps is. The host reads the remaining shares loosely ("another third or maybe 10%"), so the split is approximate. - Martin Reynolds lists four practices and calls it an operating model, not a tool. Every change is reviewable, auditable and revertible. > [!quote] [00:06:41] > It should be declarative. It should absolutely be versioned. it should be pulled automatically #### ⏱️ [00:07:26](https://www.youtube.com/watch?v=9vwISRj3V9s&t=446) Stacking benefits, and the zip-file era - Larry Garfield: you can adopt pieces and get benefits, and they compound. - A panelist recalls zip-and-unzip deployments and late-night pizza parties, then deploying on a Friday and going home. The host jokes that free food beats forcing people to return to the office. > [!quote] [00:07:26] > they stack on each other and you get more benefits the more pieces you put together. > [!quote] [00:09:39] > Don't force people to return to office. Just give them free food. #### ⏱️ [00:09:39](https://www.youtube.com/watch?v=9vwISRj3V9s&t=579) Poll 2 results and Zalando's starting point - About two-thirds: AI opens pull requests, a human reviews each. About 22%: some merge with light or no review. A smaller group reports no AI pull requests yet. - Shyamala: Zalando is above average in coding-agent use, and the volume looks like Dependabot with higher inflow. The platform predates AI and rests on a heavy GitOps investment. > [!quote] [00:10:24] > AI opens PRs and some get merged with light or no review #### ⏱️ [00:11:53](https://www.youtube.com/watch?v=9vwISRj3V9s&t=713) What a day's pull request looks like at Zalando - Checks run in GitHub, four-eyes review applies, and an infrastructure team still has end-to-end tests for its Kubernetes platform, including traffic switching. - An LLM does a first-pass review and a risk-based assessment labels each pull request high or low risk. > [!quote] [00:12:37] > we have um really high confidence in our setup #### ⏱️ [00:13:21](https://www.youtube.com/watch?v=9vwISRj3V9s&t=801) Shyamala's contention: trusting machines like humans - The risk label is only as good as the context the model sees. Automation bias is the stated worry: a low-risk label from a machine may stop people checking. - The mitigation named is a smaller incident radius with feedback at every step. - A personal note: writing code and holding judgments used to be a professional moat, and now much of that moves to agents. The questions raised are what to unlearn and what to learn. > [!quote] [00:13:21] > sometimes a low-risk PR is actually not low risk #### ⏱️ [00:14:51](https://www.youtube.com/watch?v=9vwISRj3V9s&t=891) Larry Garfield: AI is orthogonal to GitOps - Opens with a stance on how LLMs are made (copyright, labor, environment) and calls LLM use expensive, destructive, illegal autocomplete. This is opinion and no source is given. - Core argument: GitOps automates declarative tasks, and a pull request gets automated tests then human review regardless of author. Oversized pull requests get sent back to be split. > [!quote] [00:15:34] > If an LLM is filing the pull request, you're going to do the same thing. #### ⏱️ [00:16:16](https://www.youtube.com/watch?v=9vwISRj3V9s&t=976) The test-writing dispute - Larry Garfield: AI should never write tests, humans write tests. - Martin Reynolds disagrees and says AI changes the pipeline at several points, and that Harness itself assesses pull-request risk internally and surfaces the code segments that matter. - The stated condition: a different model writes the tests than the one that wrote the code. > [!quote] [00:16:16] > I would also note that the AI should never ever ever write tests. humans write tests. > [!quote] [00:16:59] > you don't want the same LLM you don't want the same model writing the tests that wrote the code right #### ⏱️ [00:17:44](https://www.youtube.com/watch?v=9vwISRj3V9s&t=1064) Test generation as bottleneck, drift as a worked example - Martin Reynolds says test generation is one of the biggest bottlenecks created by the code explosion, and that customers say so too. This is a vendor-side observation, not a measurement. - Worked example: ten drifting clusters. AI can check whether one cause explains them all, in minutes where a person needs hours. > [!quote] [00:17:44] > is one of the biggest bottlenecks with the kind of code explosion side that we get from LLMs today > [!quote] [00:18:27] > doing that analysis with AI might take a few minutes doing that analysis as a human might take a few hours #### ⏱️ [00:19:10](https://www.youtube.com/watch?v=9vwISRj3V9s&t=1150) Deterministic versus non-deterministic work - Shyamala: a recent team debate asked what problems agents actually solve. Feeling is that everything improves if an LLM touches it. The counterpoint is that deterministic work needs code, not a model. - Where models help: incident response, narrowing a bigger picture, a first report. - Martin Reynolds agrees strongly and adds a cost argument. > [!quote] [00:19:55] > most of what we do can be deterministically done > [!quote] [00:20:40] > First of all, it will be really expensive for no gain whatsoever if you have a process that's deterministic that works. #### ⏱️ [00:21:24](https://www.youtube.com/watch?v=9vwISRj3V9s&t=1284) A side trip to Jev - Martin Reynolds praises Jev, a small model that makes choices from fixed answer sets, as a low-cost option for the A-or-B decisions agents make today on large models. The transcript spells the product correctly. - Martin Reynolds states the vendor is a seed company with $40 million. No source is given. This product is not a Harness product and the panel does not pursue it. > [!quote] [00:20:40] > I did see the Jev uh model that got released. > [!quote] [00:21:24] > it is a really lightweight model that's built to make decisions based on the information it's given. #### ⏱️ [00:22:08](https://www.youtube.com/watch?v=9vwISRj3V9s&t=1328) Host reframes: does a flawed change reach production? - Host: the fundamentals are unchanged but more automated gates and more variables make an already complex practice harder. The host asks how audit trails and safety nets catch a flawed AI-generated change before production. #### ⏱️ [00:23:33](https://www.youtube.com/watch?v=9vwISRj3V9s&t=1413) Shyamala: scale-up analogy and staged rollout - The AI volume increase is the same problem a team meets growing from 2 to 20 engineers: more pull requests, more merge conflicts. Team size is flat while workload rises. - Zalando rolls out in stages, runs layered alerts (a dashboard-only tier and work-hours paging so nobody is paged at night for a low tier), and relies on observability because traffic differs across clusters. > [!quote] [00:23:33] > our team size now hasn't changed uh but that the workload has uh changed. > [!quote] [00:24:16] > So when we roll out, we roll out in stages. > [!quote] [00:24:59] > we manage uh more than 250 clusters #### ⏱️ [00:25:44](https://www.youtube.com/watch?v=9vwISRj3V9s&t=1544) Revert as the standard remedy - Rule: nobody edits a cluster by hand, because that destroys auditability. Few people hold access as a last resort for unknown unknowns. - Typical fix is toggling a config or reverting the pull request. - Martin Reynolds asks how many environments sit between CI and production. Answer: four. > [!quote] [00:25:44] > 99.99% of the time the GitHubs works really great #### ⏱️ [00:26:27](https://www.youtube.com/watch?v=9vwISRj3V9s&t=1587) Environment sprawl and tradeoffs - Martin Reynolds: managing environments is a known GitOps risk, because more environments mean more merges and more versions of everything even inside one repository. - Shyamala: the non-negotiable is the hot path where customers can buy. Stages are dev, alpha, beta, production, with production clusters in each channel, and feedback at every stage guards against regressions. #### ⏱️ [00:28:40](https://www.youtube.com/watch?v=9vwISRj3V9s&t=1720) Regulated data and old practices - Larry Garfield's employer handles medical data, so it cannot deploy many times a day or copy production data to dev. Its pipeline is still being built. - Larry Garfield's point: the listed guardrails (automated testing, observability) are not new, and they are prerequisites for good GitOps. > [!quote] [00:28:40] > We can't pull production database to dev. Uh that's just illegal. > [!quote] [00:28:40] > Having really good testing, automated testing, that's not a new concept, but it really helps with GitOps. #### ⏱️ [00:30:08](https://www.youtube.com/watch?v=9vwISRj3V9s&t=1808) Security statistics and re-checking artifacts - Martin Reynolds cites a study described as testing over 100 LLMs and found 45% of LLM-introduced code has security vulnerabilities, with Java the worst at about 70%. No study name is given. - Martin Reynolds says exploits are now published after exploitation (a minus 5 hour gap) and attributes that to AI. No source and no mechanism are given. - Implication drawn: re-check an artifact for new vulnerabilities between alpha and beta, not just at build time. > [!quote] [00:30:08] > something like 45% of code introduced by LLMs has security vulnerabilities in it > [!quote] [00:30:51] > something like minus 5 hours is uh the exploit exploit to publishing #### ⏱️ [00:31:36](https://www.youtube.com/watch?v=9vwISRj3V9s&t=1896) Governance overlay and deployment verification - Martin Reynolds: layer policy over the GitOps flow. No new critical or high vulnerabilities, up-to-date SBOM, verified artifact on every deploy. - Verification compares each deployment with previous ones. Pattern matching can flag a deploy that uses more CPU and memory even when tests pass. Martin Reynolds calls this closer to machine learning than to an LLM. > [!quote] [00:32:20] > is this deployment good compared to all my previous deployments? #### ⏱️ [00:33:45](https://www.youtube.com/watch?v=9vwISRj3V9s&t=2025) AI amplifies the setup it lands in - Shyamala: a good setup gets amplified, and so does a bad one. Fundamentals return in different forms. - The host pivots to why companies do not adopt GitOps, with a joke that budget appears when a project is labeled AI. > [!quote] [00:33:45] > if you had already a really good setup, um then it amplifies it and if you have really bad setup, > [!quote] [00:34:29] > if you want something budgeted you just say you need it for AI #### ⏱️ [00:35:15](https://www.youtube.com/watch?v=9vwISRj3V9s&t=2115) Adoption barriers: benefit gap and estate variety - Martin Reynolds: classic CI/CD already automates delivery, so the case for GitOps needs a visible gain over the status quo. Easy on greenfield, hard across thousands of services. - Not everyone deploys to Kubernetes. Martin Reynolds names customers on Windows VMs, ECS and Lambda. > [!quote] [00:36:01] > there needs to be a like a benefit gap > [!quote] [00:36:44] > we have customers who are, you know, still deploying to Windows VMs. #### ⏱️ [00:37:28](https://www.youtube.com/watch?v=9vwISRj3V9s&t=2248) Cost, fear and trust - Larry Garfield: fear of a big, complex system with many moving parts, the real cost of running it unless you buy a platform, and the argument that what works now is enough. - The deeper barrier: distrust of your own tests, checks and reviewers makes frequent deployment frightening. > [!quote] [00:38:14] > if you don't trust your test suite, if you don't trust your checks, if you don't trust your pull request reviewers, #### ⏱️ [00:38:59](https://www.youtube.com/watch?v=9vwISRj3V9s&t=2339) Business case and sleeping well - Shyamala: engineers sit between user and business needs, so the case must name customer and business benefit. The personal payoff stated is fewer night pages. > [!quote] [00:38:59] > so much uh peace that I can actually like sleep well at night #### ⏱️ [00:39:42](https://www.youtube.com/watch?v=9vwISRj3V9s&t=2382) Survey figures and auditing the AI itself - Martin Reynolds says survey figures went into the chat in the chat: 81% say GitOps improves auditability, 78% say it prevents drift, 63% say it improves security. The survey is not named. - Larry Garfield: audit must also record which model was used, under what identity, in which pipeline step, including when a model approved a low-risk pull request. > [!quote] [00:39:42] > 81% of people say GitOps improves auditability and 78% say uh it prevents drift. Only 63 says it improves security. > [!quote] [00:39:42] > what was the identity that they were using when they ran that should all be part of that audit > [!quote] [00:40:24] > what credentials it was running under, and where that sat in that pipeline #### ⏱️ [00:41:07](https://www.youtube.com/watch?v=9vwISRj3V9s&t=2467) Metrics: start from the business - Shyamala: measure customer and business health, burnout and firefighting, time to market and feedback loop speed. A platform team exists to give product engineers time back. DORA and developer-experience frameworks follow, and AI amplifies platform engineering without replacing it. > [!quote] [00:41:07] > there is a team that's burnt out, firefighting #### ⏱️ [00:43:19](https://www.youtube.com/watch?v=9vwISRj3V9s&t=2599) Lead time from pushed pull request to production - Larry Garfield: measure the time from push to production. If it is a month, there is a problem. A day or two to a push-button ship-ready state shortens feature turnaround by removing accidental complexity. > [!quote] [00:43:19] > How long does it take to get from I've pushed it up to it's on production. #### ⏱️ [00:44:47](https://www.youtube.com/watch?v=9vwISRj3V9s&t=2687) Map metrics to what the business cares about - Martin Reynolds: if the business wants speed, track lead time. If it wants stability, track time to recover and rollback percentage. - Trend beats snapshot: show getting faster without getting riskier, or more reliable without fewer releases. Businesses value predictability. > [!quote] [00:44:47] > understanding what the business cares about and mapping that to a metric that you can track > [!quote] [00:45:32] > the trend of what you're doing matters more than what the real time is #### ⏱️ [00:46:15](https://www.youtube.com/watch?v=9vwISRj3V9s&t=2775) Closing - Martin Reynolds ends with a correlation claim and the host asks everyone to keep recording DORA metrics. The panel moves to the LeadDev Slack at 00:46:58. The video runs to 47:16. > [!quote] [00:46:15] > people who have better GitHops maturity generally have better Dora metrics > [!quote] [00:46:15] > and let's all keep recording Dora metrics even if it's not corporately sponsored anymore because they are important. ## 📚 References The transcript itself contains no links; the following come from the video description or are primary sources for named entities. The video description is empty, so none of these links come from it. Every entry below is a primary source for an entity named in the panel, and no link was fetched or verified during this analysis. - [Source video: Harness - What breaks when AI opens pull requests](https://www.youtube.com/watch?v=9vwISRj3V9s), LeadDev, 47:16, uploaded 2026-10-20. - [LeadDev](https://leaddev.com): publisher and host. The panel points to its Slack community and a panel Q&A channel (00:00:05). - [Harness](https://www.harness.io): the partner named at 00:00:05 and the employer of Martin Reynolds. - [Zalando](https://www.zalando.com): employer of Shyamala per the host's questions about Zalando (00:10:24 and 00:11:53). The caption spellings are Zando and Zelando. - [OpenGitOps principles](https://opengitops.dev): the four-principle definition (declarative, versioned, pulled automatically, continuously reconciled) that matches Martin Reynolds' list at 00:06:41. The transcript does not name this source. - [DORA](https://dora.dev): the delivery metrics research program named at 00:42:36 and 00:46:15 (caption: Dora). - [Dependabot](https://docs.github.com/en/code-security/dependabot): named at 00:11:08 as the comparison for bot-opened pull requests. - [Upsun](https://upsun.com): probable current name of Platform.sh, where Larry Garfield worked in developer relations (caption: Upson). Probable correction, not confirmed by the transcript. - Statistics with no source in the transcript: above 90% adoption, under half of production apps, 16% following four or more practices (host); 45% LLM vulnerability rate, about 70% for Java, minus 5 hours exploit-to-publish, and 81%, 78%, 63% survey percentages (Martin Reynolds). Find the underlying studies before reusing any of these. ## 🗣️ Paraphrase The panel asks whether AI-generated pull requests change GitOps, and the speakers answer in three registers. Larry Garfield says the question is mostly beside the point. A pull request passes through automated tests and a human review, and a team that would reject an oversized human pull request should reject an oversized AI one. Martin Reynolds says the pipeline changes because AI can now do work at the places where a person cannot keep up: scoring the risk of a pull request, generating tests (with a second model, not the one that wrote the code), reading ten drifting clusters at once, and comparing a new deployment against the last fifty. Shyamala speaks from the operating end. At Zalando, review is two-person, tests are end-to-end, rollout is staged across four environments, and observability covers more than 250 clusters. That setup absorbs the extra volume, but the team is the same size as before and the workload is higher. Shyamala distrusts the LLM risk label because a model cannot see the context it lacks, and wants deterministic work to stay in deterministic code, with models reserved for open-ended tasks like first-pass incident reports. On why people do not adopt GitOps, the answers are a missing benefit gap against existing CI/CD, estates that are not Kubernetes, fear of a complex system, and lack of trust in your own tests and reviewers. On proving the investment, the panel agrees on starting with the business: lead time from pushed pull request to production, time to recover, rollback percentage, and the trend over time. The most repeated factual claims are unsourced. ## 📰 Newsletter **What breaks when AI opens your pull requests? Your reviewers, first.** LeadDev's 47-minute GitOps panel, made in partnership with Harness, lands on an unglamorous answer. The gates you already trust (automated tests, staged rollout, observability, revert-by-pull-request) still work, and a Zalando engineering leader says the 250-plus-cluster platform absorbs the volume. What does not scale is the person reading the diff. Team size stays flat while the workload grows. Three takeaways for anyone running a WordPress release or review pipeline. First, do not let the same model write the code and the tests that judge it. Second, keep deterministic checks deterministic, and spend model budget on judgment calls like incident first reports. Third, measure lead time from push to production and report the trend, not a snapshot. Read the statistics with care. Above 90% GitOps adoption, 45% vulnerability in LLM code and the survey percentages all arrive with no named source, and one panelist works at the partner company. Links: [the video](https://www.youtube.com/watch?v=9vwISRj3V9s), [OpenGitOps](https://opengitops.dev), [DORA](https://dora.dev). ## ✅ Action items #### DevRel - Developer advocacy: write a talk or post that separates what stays the same for AI pull requests (tests, review, revert) from what changes (reviewer load, risk labels), using the two poll results as the hook and citing the polls as unsampled. - Education: run an office-hours session on reviewing AI pull requests, using the rule that the model that wrote the code does not write its tests (00:16:59). - Developer experience: add to the contribution docs a rule for oversized pull requests, whoever or whatever authored them, and a line saying how AI authorship is disclosed. - Developer marketing: do not repeat 45%, minus 5 hours, 81%, 78% or 63% in any material until the underlying study is named and checked. - Community management: record when a recommendation comes from a vendor employee on a vendor-partnered panel, and say so alongside it. - Internal/external/cross-project: collect each project's current answer to how many reviewers see an AI pull request before merge, so cross-project numbers replace the panel's two polls. #### OSPO - Policy and governance: write a policy on AI-authored pull requests covering disclosure, required review, and whether AI may approve a low-risk change. The panel only discusses the last point as an audit need (00:40:24). - License compliance: no direct action from this source; nearest relevance: the panel's one remark on copyright (00:15:34) is an opinion without a legal analysis. - Security management: add a re-scan step between environment promotions, as Martin Reynolds describes at 00:30:51, and test whether it finds anything the build-time scan missed. - Software inventory: keep an SBOM freshness target and a verified artifact on every deploy (00:32:20), if the project ships binaries. - Developer support: document which model, under what identity, wrote or approved a change (00:39:42), so audits can answer it. - Community engagement: no direct action from this source. - Strategic alignment: pick one business-facing metric (lead time or time to recover) and report its trend quarterly (00:44:47 to 00:45:32). #### Community - Maintainers: protect reviewer capacity. Cap pull-request size and send oversized ones back to be split (00:15:34), and track reviewer hours per merged change as AI volume rises. - Contributors: state in onboarding docs that AI-assisted pull requests face the same tests and review as human ones. - Sponsors: if a vendor sponsors a panel or doc, label it sponsored at the top, as the host did at 00:00:05. - Ecosystem: share one cross-project drill: revert a bad pull request end to end and time it. #### WordPress - Core: no direct action from this source; nearest relevance: the review-load problem applies to any core ticket queue that receives AI-assisted patches. - Core Performance: no direct action from this source; nearest relevance: the deployment-versus-history comparison for CPU and memory (00:32:20) maps to performance regression checks. - Core AI: write guidance that deterministic checks stay deterministic and models handle judgment (00:19:10 to 00:20:40), and that the model writing a patch does not write its tests. - **Core Program Team:** if release-cycle retrospectives run, add reviewer load per release to the agenda, since the panel's main shared finding is that reviewers are the constraint. - Mobile: no direct action from this source. - Playground: no direct action from this source; nearest relevance: a staged environment for testing a revert fits the panel's four-environment pattern. - Community: no direct action from this source; nearest relevance: meetups could run the panel's two polls on local teams to replace the unsampled audience numbers. - Training: add a lesson on reviewing AI-authored changes: same gates, higher volume, separate model for tests. - Marketing: no direct action from this source. - Support: no direct action from this source. - Design: no direct action from this source. - Documentation: document the revert path for a bad merge, since the panel treats a pull-request revert as the default remedy (00:25:44). - Photos: no direct action from this source. - TV: no direct action from this source. - Polyglots: no direct action from this source. - Accessibility: no direct action from this source; nearest relevance: accessibility checks belong in the automated gates every pull request passes. - Meta: no direct action from this source; nearest relevance: audit logging of AI identity and credentials (00:39:42) for any automation that opens pull requests against .org repositories. - CLI: no direct action from this source. - Hosting: no direct action from this source; nearest relevance: layered alerting that pages at work hours for low-tier signals (00:24:16) is a pattern for host support teams. - Tide: no direct action from this source. - Openverse: no direct action from this source. - Themes: no direct action from this source; nearest relevance: review-queue throughput when AI-assisted submissions rise. - Plugins: no direct action from this source; nearest relevance: plugin review queues face the same reviewer-load problem, and a separate-model rule for generated tests applies to submitted plugin test suites. #### The WPOCC - Programs: no direct action from this source; nearest relevance: if The WP Open Community Collective funds maintainers, reviewer hours are the scarce resource the panel identifies. - Projects: no direct action from this source. - Funding experiments: no direct action from this source; nearest relevance: pair any AI-tooling grant with a published lead-time or time-to-recover trend, per 00:44:47. ### FAIR - Technical independence: no direct action from this source; nearest relevance: the continuous reconciliation idea (00:06:41) fits local verification of signed metadata. - Policies and covenants: require a log of which model and identity produced or approved a change (00:39:42). - Productized governance artifacts: add a reviewer-capacity line to any operator scorecard. ## 🧪 Hypotheses 1. In teams that keep reviewer headcount flat, median time-to-first-review for pull requests rises within 3 months of adopting coding agents, even when pull-request size caps are in place (testing Shyamala's 00:23:33 observation). 2. Tests generated by a different model than the one that wrote the code catch more seeded defects than tests generated by the same model (testing the 00:16:59 rule). 3. An LLM risk label of low risk has a false-negative rate (labeled low, later caused an incident) that exceeds the human-assigned low-risk rate (testing the 00:13:21 concern). 4. Teams that report higher GitOps practice counts have shorter lead times, and the relationship holds after controlling for team size and deployment frequency (testing the 00:46:15 correlation claim). 5. Re-scanning an artifact at each environment promotion finds new vulnerabilities in more than 5% of promotions that span more than 24 hours (testing the 00:30:51 argument). 6. Replacing a deterministic check with a model call raises cost per decision with no accuracy gain on the same input set (testing 00:20:40). ## 📊 Methodology - **H1.** Pull 12 months of review telemetry from 10 or more repositories. Mark each pull request AI-assisted or not using author trailers or bot accounts. Compare time-to-first-review and reviews per reviewer per week before and after agent adoption, with a difference-in-differences design against repositories that did not adopt. - **H2.** Take 50 functions with known seeded defects. Generate tests with the same model and with a different model, run both suites against the defects, and compare kill rates. Pre-register the comparison and run each condition across three seeds. - **H3.** Collect labeled pull requests from a team that uses an LLM risk label. Join labels to incident records over 6 months. Report false-negative rates for the model label and for human-assigned labels on the same pull requests, with confidence intervals. - **H4.** Use the public DORA benchmark data or an internal survey that asks for the number of GitOps practices followed and lead time. Fit a regression with team size and deploy frequency as covariates. State clearly that this tests association, not direction. - **H5.** Log a vulnerability scan at each promotion for 20 services over 90 days. Count promotions where a new critical or high finding appears between stages, and report the share. - **H6.** For 5 existing deterministic checks, build an LLM version, run both on 1,000 historical inputs, and compare accuracy and cost per decision from provider invoices. ## ❓ Questions - Which studies produced the 45% vulnerability figure, the Java figure of about 70% and the minus 5 hour exploit-to-publish figure, and what do they measure? - How many audience members answered each poll, and who were they? The host reports shares and no counts. - What does the 90%-plus adoption figure count as adoption, given the host's own 16% figure for four or more practices? - Does a pull-request size cap hold when AI tools generate changes faster than reviewers can read them? - What false-negative rate makes an LLM risk label unsafe to use as a gate? Shyamala names the risk and gives no threshold. - Which model, under what identity, approved a pull request, and where does that log live in a typical pipeline? - How does an enterprise with Windows VM and Lambda estates (00:36:44) get GitOps benefits without a Kubernetes reconciler? - If AI amplifies a setup (00:33:45), what is the smallest setup that is safe to amplify? ## 🔮 Future research - **Reviewer capacity under AI load.** No source here measures review time per pull request before and after agent adoption. A longitudinal study across open-source and enterprise repositories would test the central operational claim. - **Calibration of LLM risk labels.** The panel debates trust in risk labels with no data. A labeled-outcome study would show when a low-risk label is wrong. - **Same-model versus different-model test generation.** The marking-its-own-homework rule is intuitive and untested in the source. A controlled comparison is cheap to run. - **Audit schema for AI actors.** The panel wants model identity, credentials and pipeline position logged (00:39:42). There is no shared schema for that record. - **Independent replication of GitOps survey figures.** The 81%, 78% and 63% results need a named survey, a sample and a method before they inform decisions. ## 💡 Implications - **Practice.** Keep existing gates, add reviewer-capacity limits, and separate model roles: the model that writes does not write the tests. Stage rollouts and keep revert-by-pull-request as the default remedy. - **Policy.** Decide whether an AI may approve a low-risk pull request and record the decision. Log model, identity and credentials for every automated approval. - **Security.** Re-scan artifacts at each promotion and verify every deployment against its history. Treat the 45% and minus 5 hour figures as unverified until sourced. - **Education.** Teach reviewing as a skill: how to read a large AI-generated diff, when to send it back, how to spot a wrong low-risk label. - **Funding.** Reviewer hours are the scarce resource this panel identifies. Funding that adds generation capacity without review capacity moves the bottleneck and does not remove it. - **Vendor literacy.** A partner-labeled panel with an employee of the partner is useful for operational detail and weaker for market statistics. Read the disclosure at 00:00:05 before the numbers. ## 🧠 Learning styles assessment - **Reading level estimate.** Flesch-Kincaid Grade Level computed on the transcript with a heuristic syllable counter, 7,748 words over 327 sentences: about grade 9.7. Flesch Reading Ease about 68, plain conversational English. The sentence count is approximate because spoken text lacks clean punctuation, and the heuristic can drift by a point or two. #### 👀 Visual learners - Present: the host mentions a poll popping up on screen and a Slack channel link (00:00:05). The video format is likely split-screen talking heads, which the transcript cannot confirm. - Improve: a pipeline diagram (pull request, tests, review, staged environments, reconcile, revert) would carry the discussion in one image. Show poll results as a chart with counts, not as spoken shares. #### 🔊 Aural learners - Present: the whole session is spoken discussion. Panelists interrupt each other with disagreement and humor, which helps listeners track positions. - Improve: a 2-minute closing summary of the three positions and the agreed guardrails would help listeners who joined partway. The audio is interrupted by music cues at the start and end only. #### 🗣️ Verbal learners - Present: Martin Reynolds defines GitOps in four practices (00:06:41), and Larry Garfield gives the stacking idea (00:07:26). Anecdotes (zip-file deploys, pizza Thursday) make the history concrete. - Improve: SBOM, DORA, reconciliation, four-eyes principle and drift go undefined for newer listeners. A glossary in the description would fix that, and the description is empty. #### 🧩 Kinesthetic learners - Present: concrete operating detail: a four-stage rollout, work-hours paging, revert as a pull request. - Improve: no checklist, repository or template accompanies the video. A starter checklist (gates, rollout stages, revert drill) would turn the advice into a task. #### 🧠 Logical learners - Present: clear opposing positions, a cost argument for determinism (00:20:40), a causal argument about reviewer load (00:23:33), and a metrics-to-business mapping (00:44:47). - Improve: the statistics have no method attached. A table of claim, source and sample would let the logical learner check each one. #### 🧑‍🤝‍🧑 Social learners - Present: live chat, two audience polls, and a Slack Q&A channel (00:00:05, 00:46:58). The panel format models disagreement. - Improve: no discussion prompt is offered to teams. Questions such as what share of your pull requests are AI-opened would turn viewers into participants. #### 🧘‍♀️ Solitary learners - Present: the recording can be paused and re-watched, and the transcript supports self-paced reading. - Improve: add reflection prompts: where would a reviewer queue break first in your team, and which of your checks could you not trust on a Friday? #### 🌿 Naturalistic learners - Present: no environmental content beyond one panelist's passing reference to environmental destruction in an opinion about LLMs (00:15:34). - Improve: no energy or resource data accompanies the claim, and none is offered for the AI-in-pipeline costs the panel discusses. Reporting compute and energy per automated check would serve this group. ## 📊 Accessibility assessment #### Screen reader semantics - Present: YouTube supplies landmarks and player controls, and the transcript is machine-readable text. - Improve: any on-screen poll chart or slide is not described in the transcript, and the poll results are conveyed partly by the host reading them aloud. WCAG 2.2 AA 1.1.1 (Non-text content) applies to visuals that carry information not spoken, and 1.2.5 (Audio description, prerecorded) applies to a live-recorded panel with on-screen content. #### Keyboard navigation - Present: the YouTube player is keyboard operable with visible focus. - Improve: no chapters exist, so keyboard users must scrub a 47-minute timeline. Chapters would address 2.4.5 (Multiple ways) and 2.4.6 (Headings and labels). #### Contrast and readability - Present: unverifiable from a transcript. - Improve: if poll results or captions appear on screen, they need 4.5:1 contrast (1.4.3) and enough size on mobile. Check before publishing slides. #### Text resizing and reflow - Present: the player and captions scale independently of the page. - Improve: burned-in text in the video frame does not resize (1.4.4) or reflow (1.4.10). Prefer text in the description. #### Media and captions - Present: auto-captions exist and produced this transcript. - Improve: the captions garble the panel's key terms: GitOps as githops and GitHubs, Zalando as Zando and Zelando, LeadDev as lead dev, DORA as Dora, plus an employer name rendered two ways. Speaker labels are missing, so a deaf or hard-of-hearing viewer cannot tell who speaks. Both gaps fall under 1.2.2 (Captions, prerecorded). A live panel may also trigger 1.2.4 (Captions, live) at broadcast. Human-corrected captions with speaker names are the fix. #### Forms and interactions - Present: audience polls and a chat were part of the live session. - Improve: the recording drops the poll interaction, and a pointer to the Slack Q&A channel is spoken, not linked in the empty description. Add the channel link as text. 3.3.2 (Labels or instructions) applies to the poll on the live page, which this analysis could not inspect. #### Standards reference - Primary gaps against WCAG 2.2 AA: **1.2.2** (captions inaccurate and unlabeled by speaker), **1.2.5** (no audio description for on-screen content), **1.1.1** (poll visuals not described), **1.4.4 / 1.4.10** (burned-in text), **2.4.5 / 2.4.6** (no chapters). The empty description is also the missing home for the Slack link, a glossary, and sources for the statistics. ## 🌍 DEIB assessment #### Cultural breadth - Present: panelists span fintech, automotive, fashion e-commerce and regulated health data, and the discussion names Windows VM, ECS and Lambda estates beside Kubernetes (00:36:44). The panel crosses organization sizes from a 250-cluster retailer to a team building its first pipeline. - Improve: the examples are Europe-centered and enterprise-centered. No small-team, public-sector or non-English-language case appears. #### Inclusive language - Present: no exclusionary terms found in the transcript. Panelists use first names and job roles and avoid gendered shorthand in the speech captured. - Improve: idioms such as gamechanger, got the t-shirt, and proverbial way slow comprehension for readers of English as a second language. The host's joke about robots as people (00:05:14) is light, and the host corrects it. #### Neurodiversity support - Present: the host runs a clear structure: introductions, polls, definition, AI question, guardrails, adoption barriers, metrics. - Improve: no chapters, and interruptions between panelists make speaker tracking hard. Chapters and speaker-labeled captions would help listeners who rely on structure. #### Representation - Present: this analysis does not infer gender or other identity traits from names or voices, and the transcript states none. The panel mixes a platform engineer, a developer-relations veteran and a vendor field CTO. - Improve: the panel has one vendor employee and no representative of open-source maintainers or small teams. Add a maintainer voice for balance. #### Psychological safety - Present: disagreement stays collegial. Panelists say I disagree, then give reasons, and jokes lighten the tone (pizza Thursday, blockchain before AI). - Improve: the blunt framing of LLM use as destructive and illegal (00:15:34) could make audience members who use these tools feel judged, and no moderator response follows. A moderator acknowledgment would keep the discussion open to all. ## ✅ Logical validity #### Strong reasoning - **Analogical, 00:23:33.** Shyamala compares the AI volume increase to a team growing from 2 to 20 engineers. The comparison is apt: more authors, more merge conflicts, same fixes. Clarify further by naming what differs, such as authors that never tire and write at machine speed. - **Causal and cost, 00:19:10 to 00:20:40.** If a process is deterministic and works, replacing it with a model adds cost and no benefit. Valid, and checkable by measuring cost per decision. - **Process control, 00:16:59.** A model that writes both code and tests grades its own work. The rule of using a separate model removes that dependence. Strong as a design principle. It needs a measured effect size to become a finding. - **Prerequisite argument, 00:38:14.** Frequent deployment needs trust in tests, checks and reviewers, so distrust is a barrier. The conclusion follows directly. Strengthen it by listing what evidence builds that trust. - **Metric mapping, 00:44:47 to 00:45:32.** Choose the metric that matches what the business values, and show the trend, not a snapshot. Sound, because a single day's number says little about direction. - **Automation-bias warning, 00:13:21.** A low-risk label can mislead if the model lacks context. This identifies a real failure path, and the stated mitigation (shrink the incident radius) fits it. #### Fallacies - **Unsupported statistic, 00:30:08.** Martin Reynolds: "something like 45% of code introduced by LLMs has security vulnerabilities in it." No study, date or method is named. Fix: name the study, the languages, the definition of vulnerability and the date. - **Unsupported causal claim, 00:30:51.** "and and that's due to AI" attributes the minus 5 hour exploit-to-publish gap to AI with no evidence for the mechanism. Fix: cite the data and the mechanism, or say it is an opinion. - **Correlation presented as consequence, 00:46:15.** "people who have better GitHops maturity generally have better Dora metrics" is a correlation. Both could stem from team quality, and the direction is unstated. Fix: say the relationship is observed and its cause is unknown. - **Self-reported survey percentages, 00:39:42.** 81%, 78% and 63% report what people say GitOps does, not measured outcomes, and the survey is not named. Fix: name the survey, the sample and the question wording. - **Absolute claim without evidence, 00:16:16.** "the AI should never ever ever write tests" is an absolute with personal distrust as its only support ("I don't trust AI to write tests"). Fix: state the failure mode and test for it. - **False equivalence, 00:15:34 to 00:16:16.** Larry Garfield treats an AI pull request as identical to a human one and says it is an orthogonal question. That ignores volume, which the host (00:22:08) and Shyamala (00:23:33) both raise. Fix: separate what does not change (gates) from what does (reviewer load). - **Genetic framing, 00:15:34.** "all LLMs are built on copyright infringement, sweat shop labor, and environmental destruction" is a claim about origin used to color later use, with no source for any of the three charges. Fix: cite each, or mark it as a stated opinion. - **Poll as evidence, 00:05:14 and 00:10:24.** Live-audience poll shares serve as a picture of the field, with no sample size, no selection note and a vague third option. Fix: report counts and treat as an illustration. - **Undisclosed-interest risk (partially disclosed), 00:16:59 to 00:32:20.** The partnership is disclosed at 00:00:05 and the Harness role at 00:04:33, but Martin Reynolds' recommendations are not marked as aligned with a vendor product category. Fix: attach a short note whenever a panelist recommends a practice the partner sells. #### Tone of voice - Collegial and informal. Panelists use contractions and jokes and disagree openly, usually with a reason. Larry Garfield is the bluntest voice (00:14:51, 00:16:16). Shyamala is reflective and personal (00:13:21). Martin Reynolds is affirming and quick to add examples, and apologizes twice for adding one more point (00:19:10, 00:21:24). The host is warm, jokes about blockchain, and steers back to the question. ## 🛡️ DARVO analysis Not applicable to this source. DARVO describes a defensive response to an accusation, and this panel contains no accusation of wrongdoing against any named person. Each prong for completeness: - **Deny:** nothing is denied. Positions are disagreements about practice, and each speaker answers with reasons. - **Attack:** no attack on a critic appears. Larry Garfield's blunt remark on LLM origins (00:15:34) targets a technology and its makers in general, not an accuser, and an argument follows it. - **Reverse victim and offender:** no panelist recasts themselves as a victim. The nearest moment is Shyamala's candid note about losing a professional moat to agents (00:13:21), which is a reflection, not a deflection. ## 🎯 Conclusion The panel's useful answer to the title is that the gates you already have are the gates AI pull requests need, and that the first thing to strain is the reviewer: team size stays flat while workload grows (00:23:33). Concrete practices worth copying are staged rollout with layered alerts, revert-by-pull-request, a different model for test generation than for code, and keeping deterministic work deterministic. Reporting lead time and its trend (00:43:19 to 00:45:32) is the clearest measurement advice. The weakness is evidence. The statistics (above 90% adoption, 45% LLM code vulnerability, a minus 5 hour exploit gap, and the 81%, 78% and 63% survey figures) have no sources in the transcript. The partnership with Harness is disclosed once, and one panelist is a Harness employee whose recommendations line up with that category. **Next steps for readers:** count the AI-assisted pull requests in your own repos and the reviewer hours they take. Add a size cap and a separate-model rule for generated tests. Find the sources behind any statistic before you reuse it. Run the lead-time measure in the Methodology section on one team for a quarter before changing process. ## 🔄 Refactor Appendix — Harness - What breaks when AI opens pull requests > Density: 2–5 Literature atoms (H3 pad), 0–3 Permanent candidates (H4 pad). > Emitted: 5 Literature, 2 Permanent. Resonance filter applied: the poll shares and the > $40 million Jev funding remark were dropped as thinly evidenced or off-topic. > > Konik claim-header rules: present tense, declarative, ≤10 words, searchable, unique. > > Atomicity: one declarative sentence per atom. "and"/"also" → split. > > Tag pool: #bmf · #addendum · #xref · #articleseed · #storystem · #share · #aphorisms ### 📚 Literature Split Pad #### [H1] AI pull requests face the same gates > [!quote] [00:15:34] > If an LLM is filing the pull request, you're going to do the same thing. You're going to have automated tests that run. **Context & use**: - 00:15:34, Larry Garfield arguing AI authorship is orthogonal to GitOps. - Keep paired with Shyamala's 00:23:33 point that volume still rises, or the atom overstates what stays unchanged. **Tags**: #xref #articleseed  #### [H2] A model should not write tests for its own code > [!quote] [00:16:59] > you don't want the same LLM you don't want the same model writing the tests that wrote the code right **Context & use**: - 00:16:59, Martin Reynolds, defending AI test generation with a condition. - A design rule, not a measured result. The source gives no effect size, so treat it as a hypothesis to test. **Tags**: #xref #aphorisms  #### [H3] AI amplifies existing delivery setup, good or bad > [!quote] [00:33:45] > if you had already a really good setup, um then it amplifies it and if you have really bad setup, **Context & use**: - 00:33:45, Shyamala, before the host turns to adoption barriers. The quote cuts off mid-sentence; the speaker finishes with the bad-setup half, which also gets amplified. - A usable test for AI-readiness: would you want your current pipeline faster? **Tags**: #xref #aphorisms  #### [H4] Distrust of your own checks blocks frequent deploys > [!quote] [00:38:14] > if you don't trust your test suite, if you don't trust your checks, if you don't trust your pull request reviewers, **Context & use**: - 00:38:14, Larry Garfield on why teams hold back from GitOps. The sentence continues: then deploying several times a week is scary. - Points to trust-building as the first adoption step, ahead of tooling. **Tags**: #xref #share  #### [H5] Panel cites an unsourced LLM vulnerability rate > [!quote] [00:30:08] > something like 45% of code introduced by LLMs has security vulnerabilities in it **Context & use**: - 00:30:08, Martin Reynolds. No study, date or method is named, and the speaker hedges with something like. - Keep as a teaching specimen for checking sources before reuse, and as a prompt to find the study. **Tags**: #xref #share #articleseed  ### 🧠 Atomic Split Pad #### [P1] Review capacity is the constraint AI exposes **Claim** (one sentence, Courtney's voice): When AI authors more pull requests, the gates stay the same but the reviewer becomes the scarce resource. **Evidence pointer**: [[#H1]] · [[#H3]] **Scope & conditions**: Holds when team size stays flat and tests, staged rollout and revert already work. Breaks if review is automated or if pull-request size is capped hard enough to hold volume down. The source reports one operator's experience at 00:23:33 and no measured review times. **Links**: - [[Reviewer load in open source]] — because extends - [[Pull request size caps]] — because applies-in AI-assisted contribution queues - [[AI does not change the gates]] — because contrasts on whether volume counts as a change  #### [P2] Separate the model that writes from the model that tests **Claim** (one sentence, Courtney's voice): A model that writes code should not also write the tests that gate it. **Evidence pointer**: [[#H2]] **Scope & conditions**: A design rule from one vendor employee, untested in the source. Applies to generated tests that gate merges, and does not apply to scratch tests for exploration. A different model reduces shared blind spots, though the source does not show that it eliminates them. **Links**: - [[Independent verification]] — because extends - [[Maker-checker split]] — because applies-in AI-assisted code review - [[Test generation by the author model]] — because contrasts on who owns the check  ### ✏️ Fleeting Queue - [F1] Find the study behind the 45% LLM vulnerability figure and the Java figure of about 70% before citing either - https://www.youtube.com/watch?v=9vwISRj3V9s&t=1808 #fleeting #inbox - [F2] Confirm Larry Garfield's current employer name, which captions spell syntacter and syntax, and the correct spelling of Shyamala's first name, from the speakers' public profiles #fleeting #inbox - [F3] Open question: how many reviewer hours does an AI-assisted pull request cost versus a human one in my own repos? #fleeting #inbox - [F4] Article seed: the partner-labeled panel, where the disclosure comes once at the start and the vendor-aligned advice shows up an hour later #fleeting #inbox #articleseed - [F5] Check whether Upson in the captions is Upsun, the renamed Platform.sh, from the company's own site #fleeting #inbox ### → Things 3 (max 2 actions) - [ ] Trace the 45% LLM vulnerability and 81/78/63 GitOps survey figures to named studies before reusing them - research, 20 min #things - [ ] Measure reviewer time per AI-assisted pull request in one active repo for a month - review-load check #things  📄 **Full text:** [[Full Document Contents/Articles/Harness - What Breaks When AI Opens Pull Requests.md|Harness - What Breaks When AI Opens Pull Requests by LeadDev]] ## Highlights - [00:14:51] is given to an LLM um or like agents um and then like you know all those kind of questions. What should I unlearn now? What should I uh learn now? So, those have been sort of like my I don't know contentions basically. >> And I'm actually going to disagree uh just to keep things interesting or uh I would argue that's well first um we need to acknowledge that and all LLMs are built on copyright infringement, sweat shop labor, and environmental destruction. So if you're using an LLM for something, you're using a very expensive, destructive, illegal autocomplete. That said, if you're using it, it [00:15:34] doesn't really change anything in your GitOps workflow, I would argue, because GitOps is automation of certain tasks using declarative code. If a human files a pull request, a human is going to review it and make sure it's right. you're going to have automated tests that run before the human gets there to make sure to uh validate the things you can validate with automated testing. If an LLM is filing the pull request, you're going to do the same thing. You're going to have automated tests that run. You're going to have a human look at it to make sure it's right. If the pull request is too large to review effectively, then it doesn't matter if it's coming from a human or from an LLM, you can push back on that and say, "No, ([View Highlight](https://read.readwise.io/read/01m5d76s6rp1sm9zsy68gsqyev)) - Note: [H1] AI pull requests face the same gates - Tags: atom/literature/h1 - [00:16:16] break it up into smaller, more digestible pieces." Um, so I I would argue that using LMS for coding doesn't really change GitOps at all because it's an orthogonal question. >> I would also note that the AI should never ever ever write tests. humans write tests. I don't trust AI to write tests. >> Okay, I have a different view on that. So, so first of all, I I think AI does change it. It doesn't doesn't I'm not really talking about AI writing the change. It's all the places uh uh and already alluded to some of these things. It's all the places that AI can help along the way. So, you know, that kind of pass of the pull request to assess it [00:16:59] for risk is something that we do too. you know we internally we assess risk of pull request and we pull out the the code segments that matter for the person to and that's a change right that is AI making a fundamental change I also think that AI can write the tests the key there though is you don't want the same LLM you don't want the same model writing the tests that wrote the code right you want it to be a different LLM because you don't want something marking its own homework that's like saying hey you know you you want to give it something different to do that. But I still believe it can write the tests and you know we know um certainly from my own experience uh and actually a lot of our customers that test generation piece ([View Highlight](https://read.readwise.io/read/01m5d76vrpx5r1tj0epjeda12q)) - Note: [H2] A model should not write tests for its own code - Tags: atom/literature/h2 - [00:29:22] monitoring, that's not new. It just is a a prerequisite for good GitOps and so on. So most of these are best practices that have been around for a long time but when combined give you a much nicer smooth workflow. Uh and again you adding AI to the mix may give you tools to add additional check steps but the basic concept of uh you know autom automate your testing and validation be able to roll back have good insight into it um that's could be old hats it isn't in practice for a lot of companies but it should be >> so I mean that's interesting I I I mostly agree however I would say, you [00:30:08] know, and I know there's a stat that basically says that, you know, something like 45% of code introduced by LLMs has security vulnerabilities in it. So, it comes with more vulnerabilities than uh humans tended to introduce doing those same coding tasks and they tested like more than 100 LLMs when they did this. And Java was the worst by the way. Java got something like 70% uh you know and so I think you're right that that some of those uh things are in place but I think you know there is levels of extra checks that that almost need to go in there. Um so you know yes you should always be doing your security scanning and your remediation and make a ([View Highlight](https://read.readwise.io/read/01m5d774311tfnw5y5jw4qdw38)) - Note: [H5] Panel cites an unsourced LLM vulnerability rate - Tags: atom/literature/h3 - [00:33:45] >> I mean, one of the things that I feel that what AI has done is basically like if you had already a really good setup, um then it amplifies it and if you have really bad setup, then you're it also amplifies like that, right? So um yeah I mean uh one of the things that I keep always coming back to like it's just the fundamentals coming back to us in different other uh ways. Yeah I think >> so we may have different opinions on the call about AI but we're all very pro GitOps but we're in a bubble or y'all are in a bubble that actually have taken the [00:34:29] leap. So what is keeping companies right now from actually adopting GitOps? What has been the push back in general? And now why why can't they just say well we're using AI we need it for AI because that's what I said if you want something budgeted you just say you need it for AI. But what is holding people back from GitOps? Sorry, I I had to laugh at the uh I can get it funded if I say it's for AI, >> but I always recommend now's the time. >> It was blockchain before. >> It was You're right. It was blockchain before. Yeah, >> I I I think some of it is is you know um ([View Highlight](https://read.readwise.io/read/01m5d76ysv45em25pbbfz6cw91)) - Note: [H3] AI amplifies existing delivery setup, good or bad - Tags: atom/literature/h3 - [00:38:14] to the ne to the next stage and there's another a second button to deploy. But if you don't trust your test suite, if you don't trust your checks, if you don't trust your pull request reviewers, then deploying several times a week is really, really, really scary. So you first need to build up that trust that you know if the tests are green and the reviewer says it's okay, then I am confident that I can push it on Friday and be okay. and you need to get to that level of confidence in your system and your team uh uh in order to be able to really leverage the benefits of a GitOps workflow. >> Um yeah, I mean I agree with uh both of [00:38:59] you and I I I think that as engineers we are always sitting between the user needs and the business needs. So you really have to sort of like explain like why would you want to invest it in such a system and what benefit that would give to the customers and to the business and um I think yeah you can make a case for it because I cannot uh think about um going back to the old ways you know like this gives me um so much uh peace that I can actually like sleep well at night and uh not being woken up by a page or something. ([View Highlight](https://read.readwise.io/read/01m5d771ac5rg7dfxwybsfmqem)) - Note: [H4] Distrust of your own checks blocks frequent deploys - Tags: atom/literature/h3