Lessons Learned From Vibe Coding Plugins - CourtneyR.dev's 2nd Brain

Lessons Learned From Vibe Coding Plugins ![rw-book-cover](https://d34adp677peecb.cloudfront.net/static/images/article1.be68295a7e40.png) 🔗 **Source:** [Lessons Learned From Vibe Coding Plugins](https://www.youtube.com/watch?v=8AULmW_V-hE) > [!note]+ Document Notes > tags: [vibe-coding, wordpress-plugins, ai-hallucination, multi-model-red-teaming, wordpress-hosting] > author: WP Minute Podcasts > source: WP Minute Podcasts (YouTube) > reference: https://www.youtube.com/watch?v=8AULmW_V-hE > date created: 2026-09-29 10:32 > > > [!info] Generated analysis — not the author's words. Sections below are machine-written > > from the captured source. Only text inside `> [!quote]` blocks is verbatim from the author. > > Do not quote or attribute any other line to the named author. > > # 🏷️ Title: Lessons Learned From Vibe Coding Plugins (August 2026 Member Meetup) >  ## 🧭 Table of contents - [📝 Summary](#-summary) - [📖 Main content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning styles assessment](#-learning-styles-assessment) - [📊 Accessibility assessment](#-accessibility-assessment) - [🌍 DEIB assessment](#-deib-assessment) - [✅ Logical validity](#-logical-validity) - [🛡️ DARVO analysis](#️-darvo-analysis) - [🎯 Conclusion](#-conclusion) - [🔄 Refactor appendix](#-refactor-appendix--lessons-learned-from-vibe-coding-plugins-august-2026-member-meetup) ## 📝 Summary - **What happened.** The WP Minute Podcasts channel posted a 54:39 recording of its August 2026 member meetup (uploaded 2026-09-28, 79 views at capture). Two hosts and one guest talk: the hosts are named "Eric Caravac" and "Matt Maderos" in the auto-captions (the description says "Eric and Matt"; the second is almost certainly Matt Medeiros, and the first surname is garbled, so verify before citing). The guest is Patrick Gallagher of GridPane (captions: "Grid Pain," "Grid Pane"). The transcript carries no speaker labels; attribution below is inferred from who introduces whom and from first-person statements. - **The claim.** Patrick argues that AI coding tools now produce working software in minutes, that small open models run on roughly $5,000 of hardware, and that hallucination rates stay high enough (92% for the model Patrick names, 30% for a 27B open model) that a person who can answer for the code still has to sit in the loop. Patrick's working method is to build in one model, then have two or three other providers red-team the result. - **The WordPress angle.** Patrick says AI site builders start with Next.js or React "0% of the time" with WordPress, offers two hypotheses (dependency surface, training-data sentiment), and concedes that WordPress is the wrong fit for a static brochure site. Patrick predicts utility-plugin demand falls while niche, support-backed products hold value. - **Why it matters for plugin security and review.** The meetup is the practitioner's view of the exact problem the Plugins team faces: more submissions written by models that "sometimes" follow security guidelines (00:26:17), reviewed by authors who may not understand the output. Patrick's control (multi-model review plus an accountable human) is a process claim, not a tested one. - **Key stakeholders.** Patrick Gallagher and GridPane (a Pillar Sponsor of The WP Minute, per the editor's note in the description), the two hosts, WordPress plugin authors and buyers, the WordPress.org Plugins and Core AI teams, hosts who move very large WooCommerce databases. - **Risks and tradeoffs.** Every benchmark number comes from slides the transcript can't show (63 on an "AGI" scale, 52 for Qwen, 92% and 30% hallucination rates). The guest sells WooCommerce hosting and a search-replace tool built in-house, and the sponsor relationship is disclosed only in the description. The WP-CLI failure claim at 00:38:02 is an assertion with no test shown. - **Bottom line.** The process advice (cross-model review, human accountability, serve a specific customer) is sensible and cheap to adopt. The quantitative claims and the causal story for why models skip WordPress are unverified in this source. ## 📖 Main content **The demo opens the argument.** Patrick shows a 14-slide deck that Claude Design produced from one prompt: Patrick told it Patrick had a talk in 30 minutes, to pull the logo from gridpane.com, and to cover benchmarks and hallucinations, and it found a specific post by Jeffrey Emanuel (captions: "Jeffrey Emanuel," spelled correctly) and built a slide around it. Patrick's point is capability, not polish: "design" is the weakest computer-use skill Patrick sees, and even that clears the bar. **Capability history, told as a personal anecdote.** Patrick studied AI starting in 1998 ("25 years too early") and dates the turn to December 2024, when OpenAI released "01 pro," which Patrick calls the first reasoning model. Before it, models "would just puke out an answer" without checking it. Patrick tested with constraint satisfaction problems (the Einstein or zebra riddle) and got a correct answer after seven minutes. Contested detail: OpenAI's o1 reasoning models shipped earlier in 2024, and o1 pro mode arrived in December 2024, so "first reasoning model that ever existed" overstates it. Patrick's own experience (the December 23rd family test) is the evidence, and it is anecdote. **Benchmarks and local models.** Patrick checks Artificial Analysis about five times a week. On the slide Patrick reads out, "quad opus 5" (Claude Opus 5, presumably) sits at 63, and Qwen 3.8 27B (captions: "Quinn") at 52, next to a model captioned "GPT 5.6 six Luna." Patrick's claim: a $5,000 home machine runs a model that beats what o1 Pro was two years ago, so businesses needn't hand data to Google, OpenAI, or Anthropic ("we are the product at this point"). Patrick then extends the point to people: Patrick's wife, a physician, should be paid for the prompts, and "actual intelligence" beats model intelligence. **Why models skip WordPress.** Two hypotheses appear. Matt offers the dependency story (00:16:10): one TypeScript file is easier to pattern-match than a PHP, frontend, backend and database stack. Patrick adds a sentiment story (00:22:41, and Matt picks it up at 00:43:07): the training data is full of people saying WordPress is old or dumb, and the models also inherit people "second-guessing themselves." Patrick also says very little of WordPress code is in training corpora and offers to explain how Patrick knows, but the transcript never returns to it. **Cost and reliability from an agent run.** On Factory.ai's Droid "missions" feature, a run had gone 34 hours 10 minutes and about $200 in tokens over 36 hours. Patrick swapped the orchestrator to Kimi K3 (captions: "Kimmy K3," from Moonshot), which looped on "and then I'll write the plan" for about 3 hours. Patrick blames training data full of humans second-guessing themselves. That causal claim has no test behind it. **Hallucination rate.** Patrick reads Artificial Analysis's Omniscience hallucination rate: a named OpenAI model ("GPT 5.6 uh soul," garbled) at 92%, Command A at 14%, Qwen 3.8 27B at 30%. Meaning per Patrick's reading: when told not to answer if unsure, the model answers anyway. Patrick ties it to "helpful helper" instructions and notes humans also guess confidently. **Method for plugin work.** Two answers to the security question at 00:26:17. First, hold two to four paid accounts and red-team code across providers: build in Claude, have OpenAI's model sanity-check it, have Grok iterate on it. Second, "you need to have a human being in the loop." Patrick's case study is a database search-and-replace tool for very large WooCommerce sites, 24 hours cut to about an hour by Patrick, 10 minutes by Patrick's CTO, and 10 seconds by a later idea, handed between Patrick and a co-founder 12 to 13 hours apart so the code is worked on around the clock. **Product judgment.** With execution cheaper, ideas matter more (00:35:35). Patrick's search-replace tool matters to "maybe 500 people," and a small site finishes with WP-CLI in two minutes. The strategic lesson is to serve a defined customer who will pay, such as golf course superintendents (two businesses at tens of millions in annual revenue, unnamed). Patrick expects utility-plugin demand to fall, while established products like Gravity Forms keep value because a company stands behind them. **Sentiment and where to be.** The last third is about negativity around WordPress (Studio 2.0 reaction in Facebook groups, WordCamp 2024's keynote) and the advice to "get in better rooms." Contested: the claim that busy, winning people don't post, so public sentiment skews negative, is a survivorship argument with no data. **Unverified numbers in this source.** 43% of the web on WordPress (also mentioned as "let's say 40%"), 150,000 GridPane sites, "0% of the time" for AI site builders, the 63, 52, 92% and 30% figures, $5,000 and $100,000 hardware costs, $200 token spend, and the golf-superintendent revenue claim. None comes with a link in the transcript. ## 🗒️ Notes #### ⏱️ [00:00:00](https://www.youtube.com/watch?v=8AULmW_V-hE&t=0) Meetup kickoff - Hosts introduce the August 2026 member meetup and the guest, Patrick from "Grid Pain" (GridPane). Topic: pros and cons of vibe coding plugins and apps. - Names are garbled: "Eric Caravac" and "Matt Maderos." The description gives only "Eric and Matt." #### ⏱️ [00:00:37](https://www.youtube.com/watch?v=8AULmW_V-hE&t=37) Sponsors and thanks - Pillar sponsors named: GridPane, Pressable ("Pressible"), JetHost ("Jet Host"), WP Engine, Hosting.com, "20I" (unclear), and Weglot ("Wiglot"). - The guest is also a sponsor, which the description states as an editor's note. #### ⏱️ [00:01:09](https://www.youtube.com/watch?v=8AULmW_V-hE&t=69) AI-made slide deck - Patrick shares the screen and opens the deck. > [!quote] [00:01:32] > this entire 14 slide slide deck, um, was one single prompt inside of Claude Design. - Patrick asked for a logo from gridpane.com and talks on benchmarks and hallucinations, with a 30-minute deadline. - The model found a specific Jeffrey Emanuel post on its own and built a slide from it. #### ⏱️ [00:03:55](https://www.youtube.com/watch?v=8AULmW_V-hE&t=235) Reasoning models breakthrough - Patrick graduated high school in 1998 and studied AI, "25 years too early." - December 2024 "01 pro" is Patrick's turning point: models that check their own answers. Patrick gave it a zebra-riddle constraint puzzle at a daughter's room and got a correct answer in seven minutes. - Flag: Patrick calls it the first reasoning model. OpenAI's earlier o1 release makes that a stretch. #### ⏱️ [00:08:07](https://www.youtube.com/watch?v=8AULmW_V-hE&t=487) Benchmark treadmill reality - Artificial Analysis is Patrick's benchmark site, visited about five times a week. The chart is under 24 hours old; Opus 5 shows 63 on a scale Patrick equates with progress toward AGI. - Patrick says every model, including ones you can run at home, out-programs any human, and cites Linus Torvalds (captions: "minus Toralds") as hedging on it. Paraphrase only; the hedge is unquoted. - Qwen 3.8 27B at 52 is the highlighted bullet. #### ⏱️ [00:11:20](https://www.youtube.com/watch?v=8AULmW_V-hE&t=680) Data privacy and value - Local models mean you don't have to give data to Google, OpenAI, or Anthropic. > [!quote] [00:10:22] > you can run for 5,000 bucks in your own home. - "We are the product," compared to Facebook. Open Evidence (a physician-facing free product with advertisers) is Patrick's example of users being underpaid for their expertise. - Patrick's thesis: real expertise beats model output, so get near "actual intelligence." #### ⏱️ [00:14:31](https://www.youtube.com/watch?v=8AULmW_V-hE&t=871) Why AI skips WordPress - Follow Jeffrey Emanuel on X; a single "smartest and most radically innovative addition" prompt is Patrick's tip. - WordPress is 43% of the web, but Patrick says little WordPress code is in training data, and AI builders start with Next.js or React. > [!quote] [00:14:44] > 0% of the time they will start with WordPress. They do not start with WordPress. - Matt (00:16:10) proposes the dependency explanation: a single TypeScript file is easier to learn than a multi-layer PHP stack. #### ⏱️ [00:16:38](https://www.youtube.com/watch?v=8AULmW_V-hE&t=998) Factory missions and costs - Factory Droid "missions": one running 34 hours 10 minutes, about $200 in tokens over 36 hours. - Patrick swapped the orchestrator to Kimi K3 (Moonshot, open source, "$100,000 worth of hardware" to run), then found it looped for about 3 hours writing "I'll write the plan." - Patrick names four or five Chinese labs at the open-model frontier. #### ⏱️ [00:19:24](https://www.youtube.com/watch?v=8AULmW_V-hE&t=1164) Hallucination rates explained - The Artificial Analysis Omniscience hallucination chart: the named OpenAI model at 92%, Command A at 14%, Qwen 3.8 27B at 30%. > [!quote] [00:19:49] > 92% of the time it'll go, okay, great. Here's the answer. And it'll just spit out a guess, right? - Explanation offered: the model is told it's a helpful helper. Patrick adds that humans also guess confidently, and that a person with computer science fundamentals is "required." #### ⏱️ [00:24:00](https://www.youtube.com/watch?v=8AULmW_V-hE&t=1440) From prompts to brain tech - Anecdote: a "fancy calculator" built right the first time by a new Anthropic model (captions say "Sonic 3.7"); Patrick wondered whether to save the prompt or the code. - Speculation that prompts won't be spoken and a brain-computer interface will predict interests. A host tries to name the company; the captions give "Near," likely Neuralink, unconfirmed. #### ⏱️ [00:25:56](https://www.youtube.com/watch?v=8AULmW_V-hE&t=1556) Building plugins with AI - A host (no speaker label; probably Eric) describes building a personal plugin with Claude. > [!quote] [00:25:34] > I can get a working prototype in 5 minutes, but to actually make it usable - The security-guidelines problem, in the host's words: > [!quote] [00:26:17] > sometimes it'll use them sometimes it won't. #### ⏱️ [00:27:07](https://www.youtube.com/watch?v=8AULmW_V-hE&t=1627) Red teaming AI code - Patrick's practical answer, aimed at "everyone": > [!quote] [00:27:00] > you should have two or three or four paid accounts with all the different providers all the different models. - Build in Claude, review with the OpenAI model, red-team with Grok, iterate. #### ⏱️ [00:27:50](https://www.youtube.com/watch?v=8AULmW_V-hE&t=1670) Humans in the loop > [!quote] [00:27:46] > you need to have a human being in the loop that actually knows whether or not this is right? - Patrick has worked on one tool for about 18 months and doesn't share it publicly. #### ⏱️ [00:28:25](https://www.youtube.com/watch?v=8AULmW_V-hE&t=1705) WordPress migration breakthrough - The problem: a URL search-and-replace in the database of a big store can take 24 hours with no progress signal. - Alternatives named: a plugin, WP-CLI, or a host's built-in tool. > [!quote] [00:29:59] > and then he took it down to 10 minutes and then I had an idea that took it to 10 seconds. - The timeline Patrick gives: December 2024 first version, every new model re-tests it, CTO takes it further about 3 months ago. #### ⏱️ [00:31:18](https://www.youtube.com/watch?v=8AULmW_V-hE&t=1878) Model handoffs and sanity checks - Patrick and a co-founder in China are 12 to 13 hours apart, so work passes back and forth daily, with at least two models checking each other. - They still have to intervene when Kimi K3 "second-guesses" itself. #### ⏱️ [00:32:29](https://www.youtube.com/watch?v=8AULmW_V-hE&t=1949) Responsibility over automation - Job security argument: a company pays Patrick because Patrick takes responsibility if a WooCommerce site breaks. > [!quote] [00:32:10] > we will be the last astronaut at at the end of the line that has a gun pointed at his head. - Patrick has a head of engineering in Dallas and a CTO in China, so Patrick can ask "but why?" - Second idea: training data has few people describing stage fright, so models miss that human behavior. #### ⏱️ [00:35:35](https://www.youtube.com/watch?v=8AULmW_V-hE&t=2135) Ideas versus execution now - A host compares the human role to a general of an army: someone senior has to know what they're doing. - Patrick: the VC cliché "execution is everything" no longer holds cleanly, because execution can be farmed out. The joke product is a combined radio and urinal umbrella; the point is that users must want it. #### ⏱️ [00:36:57](https://www.youtube.com/watch?v=8AULmW_V-hE&t=2217) Serving a real niche > [!quote] [00:36:34] > It's only important to like maybe 500 people on the planet. - Small sites finish a search-replace with WP-CLI in about 2 minutes. Big stores need it correct, not just fast. - Unverified: Patrick says no existing tool was good enough to benchmark against, and that WP-CLI mishandles URLs in nested serialized strings. No test is shown. #### ⏱️ [00:39:04](https://www.youtube.com/watch?v=8AULmW_V-hE&t=2344) AI and plugin market shift - Utility plugins lose demand: a person can ask a model to add the small behavior instead of hunting the directory. - Established plugins (Gravity Forms named) stay because thousands of users report bugs and a company fixes them. - Golf course superintendents: two businesses at tens of millions, names withheld. #### ⏱️ [00:43:25](https://www.youtube.com/watch?v=8AULmW_V-hE&t=2605) WordPress negativity and reality - Matt asks whether criticism of WordPress in training data steers models away. - The Studio 2.0 anecdote (00:44:34): a host posts about agentic building in a Facebook group and gets "this still sucks" replies. > [!quote] [00:46:04] > WordPress is not a good idea for a lot of things. #### ⏱️ [00:46:15](https://www.youtube.com/watch?v=8AULmW_V-hE&t=2775) When WordPress is wrong - A brochure site that never changes is a poor WordPress fit; Patrick expects more people to skip WordPress. - Counter-anecdote: people vibe-code a quilting site, then bring it back to WordPress because there are "18 million videos" of help. (The figure is spoken as hyperbole.) - Matt Cromwell sends greetings; the theme is that successful, busy people stay quiet. #### ⏱️ [00:50:29](https://www.youtube.com/watch?v=8AULmW_V-hE&t=3029) Get in better rooms > [!quote] [00:50:25] > Get in better rooms with better people - In-person events matter more, Patrick says, because online sentiment skews negative. - Patrick describes staying focused after WordCamp 2024 and keeps delivering to customers. #### ⏱️ [00:52:55](https://www.youtube.com/watch?v=8AULmW_V-hE&t=3175) Wrap up and where to connect - A host says loud voices are not correct voices, and thanks the guest. - Contact: X at the GridPane handle, or the VideoAsk widget on gridpane.com (Patrick says Patrick sees about 95% of submissions). Topics welcome: WooCommerce, agency owners, parenting, golf, not politics. - Close: newsletter signup at the WP Minute site (the captions give "thewpmin.com/subscribe," so confirm the domain). ## 📚 References The transcript itself contains no links; the following come from the video description or are primary sources for named entities. URLs for named entities are from general knowledge and were not fetched in this session. - [Lessons Learned From Vibe Coding Plugins (August 2026 Member Meetup)](https://www.youtube.com/watch?v=8AULmW_V-hE) — the source video, 54:39, uploaded 2026-09-28. - [The WP Minute](https://thewpminute.com) — publisher; the spoken subscribe URL is garbled in captions, so confirm. - [GridPane](https://gridpane.com) — the guest's company and a Pillar Sponsor per the editor's note; the VideoAsk widget is described at 00:53:20. - [Artificial Analysis](https://artificialanalysis.ai) — benchmark and Omniscience hallucination charts Patrick reads from at 00:08:07 and 00:19:24. The exact chart version isn't linked, so Patrick's numbers can't be checked from the transcript. - [Factory](https://factory.ai) — Droid and the "missions" feature discussed at 00:16:38. - [WP-CLI search-replace](https://developer.wordpress.org/cli/commands/search-replace/) — the tool Patrick says mishandles nested serialized URLs (00:38:02). Check the documented serialized-data handling before repeating the claim. - [WordPress Plugin Handbook: security](https://developer.wordpress.org/apis/security/) — the guidelines a host says models follow "sometimes" (00:26:17). - [Detailed plugin guidelines](https://developer.wordpress.org/plugins/wordpress-org/detailed-plugin-guidelines/) and [Plugin Check](https://wordpress.org/plugins/plugin-check/) — the review baseline for the Plugins-team action items below. - [WordPress Studio](https://developer.wordpress.org/studio/) — the local environment behind the "Studio 2.0" thread at 00:44:34. - Named people and products with no URL in the source: Jeffrey Emanuel (post cited at 00:03:01), Linus Torvalds (garbled "minus Toralds"), Matt Cromwell, OpenEvidence ("Open Evidence"), Kimi K3 from Moonshot ("Kimmy K3"), Qwen 3.8 27B ("Quinn"), Command A, Gravity Forms, Claude Design. Model names such as "GPT 5.6 six Luna" and "GPT 5.6 uh soul" are garbled; confirm the real names on the Artificial Analysis pages. ## 🗣️ Paraphrase Patrick Gallagher opens by proving a point with a prop: a 14-slide deck built from one prompt in Claude Design. From there Patrick tells the story of why Patrick believes AI changed in December 2024, when a reasoning model solved a logic puzzle a daughter's earlier tools kept botching. Patrick reads two charts, one showing a 27B open model near a frontier model, another showing that the frontier model still guesses 92% of the time it should abstain. The two hosts steer toward WordPress. Why do site builders start with Next.js or React and never WordPress? Matt guesses the dependency surface. Patrick adds the sentiment story: models learn from a web that says WordPress is dead, and from people who second-guess themselves in public. Patrick then goes on record that WordPress is the wrong tool for a brochure site. The plugin advice is short. Build in one model, then have other providers review the result. Keep a person in the loop who can say whether the output is right, because someone still has to be "the last astronaut" who answers for a failure. Patrick's example is a database search-and-replace tool cut from 24 hours to 10 seconds across a founder-and-CTO handoff. Patrick then argues its market is about 500 people, that utility plugins will lose demand, and that selling to a narrow group with real pain is the way to build a business. Patrick closes on staying away from negativity and being in better rooms. ## 📰 Newsletter **The plugin that took five minutes and the plugin that took eighteen months.** At the WP Minute's August member meetup, GridPane's Patrick Gallagher spent 54 minutes on the gap between those two. The prototype-to-product rule: get a model to write it, then have two or three other models tear it apart, then have a human who understands the code say yes. If you review plugins, that last step is your job title. Two numbers stuck. Patrick reads a hallucination chart where one frontier model answers 92% of the time it should have abstained, and Patrick says a 27B open model you can run for about $5,000 still guesses 30% of the time. Patrick also says AI site builders pick WordPress "0% of the time." None of those figures come with a link in the video, so treat them as prompts for your own testing, not as citations. Worth your time if you build or review plugins: the case study of a 24-hour URL search-and-replace shrunk to 10 seconds, and the argument that utility plugins lose ground while niche, supported products keep it. Patrick is also candid that GridPane is a sponsor of The WP Minute (the description says so), which is worth remembering while Patrick explains why human accountability is where a hosting company earns its money. Links: [the video](https://www.youtube.com/watch?v=8AULmW_V-hE), [Artificial Analysis](https://artificialanalysis.ai), [Plugin Check](https://wordpress.org/plugins/plugin-check/), [WP-CLI search-replace](https://developer.wordpress.org/cli/commands/search-replace/). ## ✅ Action items #### DevRel - Developer advocacy: turn 00:27:00 into a demo, one plugin built in model A and reviewed by models B and C, with the diff of findings published and each finding labeled real, false positive, or missed. - Education: run a workshop titled "Prototype in 5 minutes, ship in 5 weeks," using the 00:25:34 quote as the framing and Plugin Check output as the exercise. - Developer experience: publish a security checklist that models can read (nonce, capability, escaping, sanitization) and test whether it changes model output, given the "sometimes it'll use them" report at 00:26:17. - Developer marketing: don't repeat the 92%, 30%, or "0% of the time" figures in Pantheon or community material without a linked benchmark. Cite the mechanism or run the test. - Community management: add a note to meetup recaps that a guest is also a sponsor, matching the editor's note the description already carries. - Internal/external/cross-project: ask GridPane and WP Minute whether the slide deck and chart sources can be shared, so numbers are checkable. #### OSPO - Policy and governance: write a rule for AI-assisted contributions to any owned plugin: name the tool used, state who reviewed it, and require that the reviewer can explain the change. - License compliance: check whether model-generated code carries license or provenance obligations in your projects, since the meetup never addresses it. - Security management: add "generated by a model, reviewed by a different model" as a required step, not a substitute, alongside human review for anything touching auth, payments, or serialization. - Software inventory: tag which plugins and internal tools began as vibe-coded prototypes, so audits prioritize them. - Developer support: office hours for plugin authors on reading model output critically; use the search-replace serialized-string edge case (00:38:02) as a worked example once it is reproduced. - Community engagement: no direct action; nearest relevance is sponsor disclosure norms in member content. - Strategic alignment: track the share of new plugin submissions that state AI assistance, and the rejection rate for that group versus others. #### Community - Maintainers: require a short "how I verified this" note on AI-assisted pull requests, because maintainers can't tell a 5-minute prototype from a 5-week product. - Contributors: add a contributor guide section on cross-model review, with the caveat that it lowers error but is untested against real vulnerabilities. - Sponsors: when a sponsor appears as a guest, state the relationship in the video itself, not only in the description. - Ecosystem: coordinate one disclosure drill for an AI-generated plugin vulnerability, so the chain from author to reviewer to host is rehearsed. #### WordPress - Core: no direct action from this source; nearest relevance is the "why not WordPress" question, which sits with docs and tooling more than core code. - Core Performance: the 24-hour to 10-second search-replace claim is a performance story for very large databases. Ask for the method and measure against WP-CLI on a shared dataset. - Core AI: publish and test machine-readable WordPress coding and security guidance, and measure whether models that read it follow security rules more often than the "sometimes" at 00:26:17. - **Core Program Team:** flag that the meetup's "AI skips WordPress" claim and the 92% hallucination figure are unverified, so cross-team decks don't cite them yet. - Mobile: none from this source. - Playground: a Playground blueprint that loads a vibe-coded plugin next to Plugin Check output would make the "5 minutes to prototype, weeks to trust" gap visible to newcomers. - Community: meetup kit item: a 15-minute "red-team your own plugin across two models" session for local groups, with a reviewer checklist. - Training: a lesson on reviewing AI-generated PHP for the Security Trinity (sanitize, validate, escape), nonces, and capability checks, plus what a model tends to skip. - Marketing: no direct action; nearest relevance is the "43% of the web" figure, which needs a current W3Techs citation before reuse. - Support: prepare for users who vibe-code a site, then return to WordPress (00:46:48); a "moving a generated site into WordPress" guide would answer that pattern. - Design: none from this source. - Documentation: add or improve beginner docs aimed at the reader who asks a model "make me a website," so the corpus that models see includes current WordPress guidance. - Photos: none from this source. - TV: none from this source; nearest relevance is that meetup recordings like this one need corrected captions. - Polyglots: none from this source. - Accessibility: request corrected captions and speaker labels for meetup recordings (see the accessibility assessment). - Meta: study whether plugin-directory install trends show the utility-plugin decline Patrick predicts at 00:38:45. - CLI: verify or refute the claim that `wp search-replace` mishandles nested serialized URLs (00:38:02) with a reproducible fixture and, if real, file it. - Hosting: cross-host checklist for large-database search-and-replace (progress reporting, serialized data handling, rollback), since "24 hours and you don't know how far along" is the pain Patrick names. Also record that the guest's company is a sponsor and a host. - Tide: score AI-assisted plugins separately for a period and see whether findings differ; the 92% abstention-failure figure suggests they may. - Openverse: none from this source. - Themes: none from this source; nearest relevance is that the same review-throughput pressure from AI-assisted submissions applies. - Plugins: add an AI-assistance disclosure field to submissions, run Plugin Check on every review, and require that the author can answer review questions in their own words, since the guest's own control is "a human being in the loop." #### The WPOCC - Programs: none directly. If The WP Open Community Collective funds maintainer tooling, fund review capacity for AI-assisted submissions before funding generation tools. - Projects: a documentation sprint for "how to review AI-generated WordPress code" would use this meetup as a starting scenario. - Funding experiments: fund a small benchmark that tests the "0% of the time" claim and the 30% and 92% figures on WordPress tasks, published with method and data. ### FAIR - Technical independence: multi-model red teaming depends on hosted models from several vendors. Any FAIR-adjacent review tooling that copies it inherits those availability dependencies. - Policies and covenants: require that AI-assisted package releases record which model built and which reviewed, so provenance survives into metadata. - Productized governance artifacts: add an "AI-assisted, human-accountable" row to operator scorecards, matching the accountability argument at 00:32:10. ## 🧪 Hypotheses 1. AI site builders choose WordPress in fewer than 5% of "make me a website" prompts across four major tools, versus the "0% of the time" claim at 00:14:44. 2. A second model reviewing code generated by the first flags at least 30% more real security issues than the generating model self-reviewing, on WordPress plugin tasks. 3. Providing a machine-readable WordPress security checklist in the prompt raises the share of generated plugins passing Plugin Check with zero security errors by 25 percentage points or more. 4. `wp search-replace` mishandles URLs inside nested serialized strings in at least one reproducible case, as Patrick claims at 00:38:02, but the failure rate on real production databases is under 1%. 5. Within 12 months, new installs of single-purpose "utility" plugins fall faster than installs of established multi-feature plugins, consistent with 00:38:45. 6. The 92% and 30% abstention-failure figures shrink by half or more when the prompt supplies a fill-in "I don't know" option and penalizes wrong answers, on the same benchmark. ## 📊 Methodology - **Design for H1.** Write 50 site-building prompts across 5 categories (blog, store, brochure, membership, portfolio). Run each through 4 AI site builders, 3 seeds each. Record the framework chosen, and repeat with the prompt "use WordPress" removed and added. - **Design for H2.** Generate 30 plugins from 30 task briefs with model A. Have model A, then B and C, review each. Score findings against a human security reviewer's labels and against Plugin Check output. Report precision, recall, and cost. - **Design for H3.** Same 30 briefs, two arms: bare prompt versus prompt plus a checklist. Count Plugin Check errors and warnings per plugin. Pre-register the 25-point threshold. - **Design for H4.** Build a fixture database with nested serialized values containing escaped URLs at 4 nesting depths. Compare `wp search-replace` against Patrick's tool and a hand-verified expected output. If Patrick's tool isn't public, that absence is the finding. - **Design for H5.** Pull directory install and update data for 200 plugins tagged utility and 200 established plugins, before and after 2025, and compare trend slopes. - **Design for H6.** Re-run the Omniscience-style questions with and without an explicit abstain option and a penalty for wrong answers. Report abstention failure rate per condition. - **Controls.** Fix model versions and dates, publish prompts, and use the same reviewers blinded to which model generated the code. ## ❓ Questions - Which chart version and date produced 63, 52, 92%, and 30%, and what were the exact prompts when the model was "told not to answer"? - Does the multi-model red-team practice catch vulnerabilities a single review misses, or does it mostly agree with itself? - Does the "0% of the time" statement come from a systematic test or a handful of tries, and what were the tools and prompts? - What exactly does `wp search-replace` get wrong in the nested serialized-string case, and is the behavior documented? - If the reason models skip WordPress is training-data sentiment, why would the same models pick React over Vue, or Next.js over Remix, at similar rates? - How should a review team treat a plugin whose author says they "vibe coded" it and can't explain a function? - Who is accountable in the "last astronaut" model when the author is a solo hobbyist and the host, marketplace, and buyer have no relationship? - Does the hosts' position that a sponsor guest should be given wide latitude change how listeners should weigh the claims? ## 🔮 Future research - **Security regressions in AI-assisted plugins.** No dataset compares vulnerability density by generation method. A directory-wide, labeled sample would settle whether the concern is real. - **Cross-model review efficacy.** Whether models reviewing each other's code catch different bugs or share blind spots is untested for WordPress-specific issues such as capability checks and escaping. - **Training-data composition.** The claim that little WordPress code sits in training corpora and that sentiment steers choices needs an audit of public code and text distribution. - **Abstention behavior.** The hallucination rates discussed are aggregate. Per-task behavior on plugin review, where "I don't know" matters most, is a gap. - **Plugin market structure.** Effects on utility-plugin demand, support burden, and update cadence are speculative in this source and measurable with directory data. ## 💡 Implications - **Practice.** Treat a generated prototype as an unreviewed contribution. The 00:25:34 quote (a working prototype in 5 minutes, usability takes far longer) is the workflow warning to teach. - **Policy.** Plugin review needs an AI-assistance disclosure and an accountability rule: the author must be able to explain the code. Patrick's "human in the loop" is the informal version. - **Security.** Multi-model review is a filter, not a guarantee. It sits alongside static analysis, Plugin Check, and human review. A model that "sometimes" follows security guidelines (00:26:17) needs those guidelines enforced by tooling, not by hope. - **Education.** Reviewing generated PHP is now a beginner skill. Curricula need "read model output critically" alongside "write a plugin." - **Funding.** Reviewer time becomes scarcer than code. Funding that pays for review capacity buys more assurance than funding that pays for generation. - **Hosting.** Hosts that sell "we take responsibility" (00:32:10) have a commercial reason to say so. That doesn't make it wrong, but it does make disclosure necessary. - **Data privacy.** The local-model argument at 00:11:20 (don't send your data to hosted vendors) is relevant to hosts and agencies handling customer data, and needs a real cost and capability comparison before it's a policy. ## 🧠 Learning styles assessment - **Reading level estimate.** Flesch-Kincaid Grade Level, hand-estimated from the transcript's spoken register: about grade 7. Flesch Reading Ease lands near 75 for conversational speech. Long run-on sentences and filler ("um," "you know") lower the score on paper. Technical terms (constraint satisfaction, orchestrator, red-teaming, serialized strings) push comprehension to an intermediate WordPress developer audience. The figure is an estimate, not a computed score. #### 👀 Visual learners - Present: a 14-slide deck on screen, benchmark charts, and a Factory missions dashboard. - Improve: the numbers on the charts are read aloud partially. Add a captioned slide export or a description that lists each model, score, and date. #### 🔊 Aural learners - Present: the full talk is spoken, with a warm conversational pace and repeated key points. - Improve: a 3-minute audio summary of the plugin-review advice would serve listeners who drop in at the chapter list. #### 🗣️ Verbal learners - Present: vivid analogies (last astronaut, general of an army), and a clear personal anecdote for each claim. - Improve: "orchestrator," "constraint satisfaction," and "serialized strings" go undefined. One clause each would cover them. #### 🧩 Kinesthetic learners - Present: the deck-from-one-prompt demo and the zebra-riddle story invite copying. - Improve: the exact prompts, the red-team steps, and the search-replace test case aren't provided. Post the prompt and a starter checklist. #### 🧠 Logical learners - Present: a stated mechanism for the workflow (build, red-team with other models, human sign-off) and a timeline for the search-replace speedup. - Improve: the benchmark figures and the "0% of the time" claim have no method. Show the counts or the chart source. #### 🧑‍🤝‍🧑 Social learners - Present: a live meetup format with two hosts, an audience Q&A at the end, and multiple speakers reacting to each other. - Improve: no discussion prompt for viewers and no member forum link in the transcript. Add one. #### 🧘‍♀️ Solitary learners - Present: chapters let a viewer skip to the plugin-security segments alone. - Improve: add a self-check: "Which of your last three plugin commits did a model write, and who reviewed it?" #### 🌿 Naturalistic learners - Present: little explicit. The golf-course-superintendent example is a grounded, real-world niche. - Improve: the local-model argument has an energy-per-inference angle that goes unmentioned. Naming it would land with this group. ## 📊 Accessibility assessment #### Screen reader semantics - Present: YouTube supplies landmarks and a 23-chapter list, which gives screen reader users a navigable outline. - Improve: charts and slides carry data the speech only partly gives (63, 52, 92%, 30%). WCAG 2.2 AA 1.1.1 (Non-text Content) and 1.2.5 (Audio Description, Prerecorded) are not met for those visuals. #### Keyboard navigation - Present: the YouTube player is keyboard operable with visible focus, and chapters give jump points. - Improve: nothing in the video adds barriers. 2.1.1 and 2.4.7 pass via the platform. Chapter titles help 2.4.5 (Multiple Ways) in spirit, although it applies to pages. #### Contrast and readability - Present: unverifiable from the transcript. Patrick says the chart is "kind of hard to see" (00:09:37), which suggests small text on the slide. - Improve: enlarge chart text and zoom into the highlighted bullet during screen share. Risk against 1.4.3 (Contrast Minimum) and 1.4.4 (Resize Text) for burned-in slide text. #### Text resizing and reflow - Present: the player scales, and captions resize independently. - Improve: text baked into a shared screen doesn't reflow. 1.4.4 and 1.4.10 (Reflow) fail for slide content inside the frame. #### Media and captions - Present: captions and chapters exist, and the description gives a summary. - Improve: the auto-captions garble names and products: "Eric Caravac," "Matt Maderos," "Grid Pain," "Quinn 3.8," "Kimmy K3," "minus Toralds," "Wiglot," "Sonic 3.7," and the model names "GPT 5.6 six Luna" and "GPT 5.6 uh soul." That is a 1.2.2 (Captions, Prerecorded) failure in substance. There are no speaker labels in a multi-speaker meetup, which hurts comprehension, so corrected captions with speaker names are the highest-impact fix. #### Forms and interactions - Present: the VideoAsk widget on gridpane.com is described spoken only. - Improve: put the contact links in the description as selectable text. 3.3.2 (Labels or Instructions) applies to the destination pages. #### Standards reference - Primary gaps against WCAG 2.2 AA: **1.2.2** (inaccurate captions and no speaker identification), **1.2.5** (no audio description for slides and charts), **1.1.1** (chart data conveyed visually), **1.4.4 / 1.4.10** (burned-in text). Corrected captions with speaker labels are the cheapest fix. ## 🌍 DEIB assessment #### Cultural breadth - Present: mentions a co-founder in China, Chinese open-model labs, and hosts operating across time zones. The examples otherwise center on the United States. - Improve: add non-US and non-English WordPress builders. The golf-superintendent and lemonade-stand examples are US-coded, and no multilingual or non-Western case appears. #### Inclusive language - Present: mostly direct, second-person, and jargon is often explained by analogy. - Improve: idioms and violent or crude phrasing appear: "gun pointed at his head" (00:32:10), "eat another bowl of glass" (00:33:39), "punched in the teeth" (00:48:56), "shoot your foot off" (00:22:41 area), and a running joke involving a urinal and other bodily products. "Clankers" (00:31:27) is a slang term for AI tools. Swap idioms for literal statements, and drop the bodily jokes in member content. #### Neurodiversity support - Present: 23 chapters and a stable structure of claim, anecdote, hosts' question. - Improve: tangents (brain interfaces, sports analogies, stage-fright digression) and undefined jargon raise the load. A short recap slide at each chapter would help. #### Representation - Present: mentions of a physician wife, three daughters, a CTO and head of engineering, and Matt Cromwell give some people a voice by reference. - Improve: all three speakers are men and no woman speaks in the recording. The "Nana" quilting-site example casts an older woman as the person needing help. Invite a plugin reviewer or a non-male builder to the next meetup. #### Psychological safety - Present: hosts respond warmly. A host asks a security question in a non-judgmental way, and Patrick sets a boundary ("not politics"). - Improve: the passage on WordPress critics (00:48:14) frames people who post negative feedback as non-builders. That may discourage honest critique from members, so a line inviting constructive criticism would help. ## ✅ Logical validity #### Strong reasoning - **Causal and procedural, 00:27:00.** Building in one model and having others review reduces the chance a single model's blind spot survives. It's a reasonable process argument. Clarify by stating what it catches and what it doesn't, since "sometimes it'll use them sometimes it won't" applies to reviewers too. - **Accountability argument, 00:32:10.** Someone has to answer when something breaks, and models can't. The point holds for hosting and for plugin authors. Strengthen it by naming who is accountable in each case. - **Market reasoning, 00:38:45 and 00:39:28.** Utility plugins lose demand because users can now ask a model for the small behavior, while established products keep users because they carry support and thousands of bug reports. It's a coherent mechanism. To strengthen it, give directory install data. - **Concession against interest, 00:46:04.** "WordPress is not a good idea for a lot of things" raises credibility for a host in the WordPress business. - **Prototype versus product, 00:25:34.** A working prototype in 5 minutes is not a usable product. This distinction is the most transferable claim in the meetup. #### Fallacies - **Overgeneralization, 00:14:44.** "0% of the time they will start with WordPress." No test set, tools, or prompts are offered, and the claim is an absolute. *Fix:* report the number of prompts, the tools tried, and the dates. - **Unsupported quantification, 00:09:37 and 00:19:49.** "every single model that exists right now ... they are better at programming than any human alive" and the 92% hallucination figure both rest on slides the transcript can't show. *Fix:* cite the exact chart, date, and question set. - **False dilemma, 00:40:55.** "I'll just gamble it all on AI. I'll just gamble every aspect of my life" frames the only alternatives as total trust or none. Most teams use a middle path, which Patrick describes at 00:27:00. *Fix:* compare the middle path against both extremes. - **Survivorship and generalization about critics, 00:47:30 and 00:48:14.** Busy, winning people don't post, so online sentiment skews negative, and critics "don't run a podcast" or own businesses. No data is given, and some critics do run businesses. *Fix:* replace with a sample or drop the claim about who the critics are. - **Commercial interest undisclosed in the video, 00:00:37 and 00:32:10.** GridPane is a sponsor, and the argument that human accountability protects hosting revenue serves the guest. The description discloses it but the speakers don't. *Fix:* a spoken line at the start. - **Unverified causal story, 00:18:22 to 00:19:06 and 00:43:50.** A model looping on "and then I'll write the plan" is attributed to training data full of self-doubting humans, and skipping WordPress is attributed to negativity. Neither is tested, and each has simpler explanations (reward design, dependency count). *Fix:* label these as hypotheses. #### Tone of voice - Warm, storytelling, and confident, in the register of a founder talking to peers. Second person and contractions throughout, with jokes and personal detail ("I've got three daughters," 00:05:12). Confidence peaks on numbers and predictions (00:09:37, 00:40:12) and softens on process, where Patrick says "there is no perfect answer" (00:32:54). Humor grows more crude near the middle and end. Hosts are supportive and rarely challenge specifics, which shapes how much of the talk was checked live. ## 🛡️ DARVO analysis Not applicable to this source. DARVO (Deny, Attack, Reverse Victim and Offender) describes a response to an accusation, and this is a friendly member meetup with no accuser and no accused. For completeness across the three prongs: **Deny** — nothing is denied; the guest concedes WordPress is wrong for some sites (00:46:04) and that models hallucinate. **Attack** — the closest match is the passage at 00:48:14 and 00:48:56, where critics of WordPress are described as people who "sit in a glass house and chuck rock" without running businesses. That is a generalized dismissal of critics, not a rebuttal of a specific charge. **Reverse victim/offender** — no reversal appears; the WordPress community is framed as a target of negativity, but no accusation is being answered. ## 🎯 Conclusion The durable advice is procedural: prototype fast, have other models attack the result, and keep a person who can explain the code in the loop, because someone has to be "the last astronaut" when it breaks (00:32:10). For plugin security and review, that maps to disclosure of AI assistance, Plugin Check on every submission, and an author who can answer questions in their own words. The niche-product argument at 00:36:57 and 00:39:04 is a market thesis worth testing against directory data. The quantitative and causal claims don't survive scrutiny in this source. The 92% and 30% hallucination rates, the 63 and 52 scores, the "0% of the time" figure, and the WP-CLI failure claim all come without a method or link. The guest is a sponsor and a WooCommerce host whose argument (human accountability is the moat) also serves the guest's business. **Next steps for readers:** run the two-model review process on one of your own plugins and record what each model found that the other missed. Pull the Artificial Analysis chart yourself before repeating any of these numbers. Confirm host and product names (Eric, Matt Medeiros, Qwen, Kimi K3, the GPT model names) from the description or the vendors, because the auto-captions garble most of them. ## 🔄 Refactor Appendix — Lessons Learned From Vibe Coding Plugins (August 2026 Member Meetup) > Density: 2–5 Literature atoms (H3 pad), 0–3 Permanent candidates (H4 pad). > Emitted: 5 Literature, 2 Permanent. Resonance filter applied: the brain-interface speculation, > sports analogies, and sponsor list were dropped as off-thesis. > > Konik claim-header rules: present tense, declarative, ≤10 words, searchable, unique. > > Atomicity: one declarative sentence per atom. "and"/"also" → split. > > Tag pool: #bmf · #addendum · #xref · #articleseed · #storystem · #share · #aphorisms ### 📚 Literature Split Pad #### [H1] Accountable humans stay in the loop > [!quote] [00:27:46] > you need to have a human being in the loop that actually knows whether or not this is right? **Context & use**: - 00:27:46, right after the multi-model red-team answer, and the only control Patrick says can't be replaced. - Directly relevant to plugin review: an author who can't explain the code is the failure case. The claim comes from a sponsor guest with a commercial interest in accountability. **Tags**: #xref #share  #### [H2] Cross-model red teaming refines generated code > [!quote] [00:27:00] > you should have two or three or four paid accounts with all the different providers all the different models. **Context & use**: - 00:27:00, Patrick's most practical answer to the host's question about verifying security guidelines. - It is a process claim with no test behind it. Treat as a hypothesis to measure with Plugin Check and human review labels. **Tags**: #xref #articleseed  #### [H3] Model iteration cut a 24-hour task to seconds > [!quote] [00:29:59] > and then he took it down to 10 minutes and then I had an idea that took it to 10 seconds. **Context & use**: - 00:29:59, the search-and-replace case study, part of a chain from 24 hours to about an hour, 10 minutes, then 10 seconds. - The speedup is asserted, not shown. Keep it as an example of repeated model-assisted rework by a human team, and ask for the method. **Tags**: #xref  #### [H4] Frontier models guess when told not to > [!quote] [00:19:49] > 92% of the time it'll go, okay, great. Here's the answer. And it'll just spit out a guess, right? **Context & use**: - 00:19:49, read from an Artificial Analysis hallucination chart the transcript can't show. The model name is garbled in captions. - If it replicates, it's the reason to require human review of generated plugin code. Verify the chart and prompt before citing. **Tags**: #xref #share  #### [H5] AI site builders skip WordPress by default > [!quote] [00:14:44] > 0% of the time they will start with WordPress. They do not start with WordPress. **Context & use**: - 00:14:44, the opening of the "why AI skips WordPress" thread, with two competing explanations (dependency surface at 00:16:10, negative sentiment at 00:22:41). - The absolute figure has no test shown. Use as a hypothesis for Core AI and Documentation work. **Tags**: #xref #articleseed  ### 🧠 Atomic Split Pad #### [P1] Review capacity, not code, limits AI-assisted plugins **Claim** (one sentence, Courtney's voice): When a model can write the plugin, the scarce resource is a reviewer who can explain why the code is safe, so submission policy should require that person by name. **Evidence pointer**: [[#H1]] · [[#H2]] · [[#H4]] **Scope & conditions**: Holds for code that touches auth, payments, serialization, or user data. Weakens for throwaway personal tools. Depends on cross-model review being a filter, not a substitute, which this source doesn't test. **Links**: - [[Plugin review and provenance]] — because applies-in AI-assisted submissions - [[Security Trinity for plugin code]] — because extends to generated code - [[Reading vendor benchmarks]] — because contrasts on evidence: this meetup gives process advice without measured results  #### [P2] Training-data absence may explain default stack choice **Claim** (one sentence, Courtney's voice): Models that skip WordPress may be reflecting how little current WordPress code and guidance they've seen, which makes docs and machine-readable guidance a lever the community controls. **Evidence pointer**: [[#H5]] **Scope & conditions**: An untested causal claim. Competing explanations: dependency surface, negative sentiment, and benchmark habit. Fails if models pick other multi-file PHP or full-stack frameworks at similar rates. **Links**: - [[WordPress AI guidance for models]] — because extends the docs argument - [[Why AI picks Next.js by default]] — because contrasts on cause - [[Core AI evaluation sets]] — because applies-in how to test it  ### ✏️ Fleeting Queue - [F1] Confirm host and product names from the description or vendors: Eric's surname, Matt Medeiros, Qwen 3.8 27B, Kimi K3, and the two garbled GPT model names — https://www.youtube.com/watch?v=8AULmW_V-hE #fleeting #inbox - [F2] Pull the Artificial Analysis Omniscience chart and confirm 92%, 14%, and 30% before quoting — https://artificialanalysis.ai #fleeting #inbox - [F3] Test whether `wp search-replace` mishandles nested serialized URLs as claimed at 00:38:02, using a fixture database #fleeting #inbox - [F4] Article seed: "Prototype in 5 minutes, trust in 5 weeks" for plugin authors, built on the 00:25:34 quote #fleeting #inbox #articleseed - [F5] Open question: do AI site builders really pick WordPress 0% of the time? Run the 50-prompt test from the Methodology section #fleeting #inbox - [F6] Track utility-plugin install trends against Patrick's 00:38:45 prediction, using directory data #fleeting #inbox ### → Things 3 (max 2 actions) - [ ] Verify the 92%, 30%, and "0% of the time" figures against Artificial Analysis and a small site-builder test before citing this meetup — research, 30 min #things - [ ] Draft an AI-assistance disclosure and reviewer-accountability proposal for the Plugins team, using the Action items section — plugin review #things  📄 **Full text:** [[Full Document Contents/Articles/Lessons Learned From Vibe Coding Plugins.md|Lessons Learned From Vibe Coding Plugins by WP Minute Podcasts]] ## Highlights - [00:14:44] that makes up WordPress is actually part of the training corpus that's in all of these large language models. Um and if you want I can tell you how I know this to be a fact. Um but it turn but it turns out that that um if you go to Claude or you go to any one of these models and you say hey make me a website. you go to you go to replet or um lovable and you say make me a website 0% of the time they will start with WordPress. They do not start with WordPress. They start with Nex.js. They start with React. They start with they start with things that are not PHP. Yeah. Um and and this is sort of good, sort of bad. Like um I think I mean [00:15:28] actually what what's you guys' sense on on this? like it you you probably talk to a lot more WordPress product owners than me. Um how how deep down the rabbit hole would you say a lot of them have [clears throat] gone in this context? >> That's a good one. You know, I I have never thought of it that way, but I also didn't assume that it would use WordPress unless I specifically asked. Um you know, I maybe I don't know, Matt, what do you what you think? But maybe it's it's the dependencies of WordPress. um you know if you're using Nex.js JS or something like that, you're just kind of using that one package, right? And and you're not >> Yeah, I I'd assume the same thing. the ([View Highlight](https://read.readwise.io/read/01m3pspq45dzrvsqq03ezzbfss)) - Note: [H5] AI site builders skip WordPress by default - Tags: atom/literature/h3 - [00:19:06] training? It's a lot of people secondguessing themselves. It's just an enormous amount of people kind of playing house, right? And um and I think that that uh it's clearly in in these models, right? And it's like so so one of the things that um that I think is uh really scary that's still true. So you can get artificial analysis also. This is the artificial analysis omniscience hallucination rate. Okay. And so these are all of the most impressive models on the face of the planet. And this is the rate at which they hallucinate an answer. Okay? And you'll notice this [00:19:49] all these black bars over here. These are all of the OpenAI uh frontier models right now. GPT 5.6 uh soul is right here at 92%. Okay, this is the most impressive model arguably of the front runners of the Coke and Pepsi of Anthropic and and Open AAI. This is their most impressive model. And if you tell it, if you don't know the answer, do not answer. 92% of the time it'll go, okay, great. Here's the answer. And it'll just spit out a guess, right? Like it knows that it's not supposed to do that and it still does it anyway. And the reason it ([View Highlight](https://read.readwise.io/read/01m3pspmn0r8kne3fhgprhabxv)) - Note: [H4] Frontier models guess when told not to - Tags: atom/literature/h3 - [00:26:17] to um also understand that the methods that these models use may not always be the best method. Um there are certain things like even with WordPress you can tell it you know well here are the security guidelines >> use those guidelines and sometimes it'll use them sometimes it won't. How how have you kind of navigated that and just sort of verifying what the model's doing and so that it's something that you know you can safely use and and maybe have other people use. >> Yeah. So um so it's very good question. So the the short answer to that is um the or the the answer that's most useful [00:27:00] to everyone I would say is you should have two or three or four paid accounts with all the different providers all the different models. Yeah. And what you want to be doing is you want to be based red teaming whatever code you create you want you know let's say you build it in anthropic in claude you build version one inside of claude you then would want to have open AAI 5.6 GPT 5.6 Saul basically sanity checking that code right and then you could have gro 4.6 six also, you know, um red teaming that code and basically creating iterations um and refining and refining and refining. Um ([View Highlight](https://read.readwise.io/read/01m3pspfkfqagmecf8ny8k2nwx)) - Note: [H2] Cross-model red teaming refines generated code - Tags: atom/literature/h2 - [00:27:46] but then you know unfortunately the um the the sad fact at the end of the day is it still just comes back to um you need to have a human being in the loop that actually knows whether or not this is right? Whether this is brilliance or not. Yeah. Um and and um so a hyper specific example of this is I've been working on something for about the last 18 months that's way off in the weeds um that I I don't actually care sharing it. Yeah. So, so if you are a hosting company for example and you have [00:28:29] really big WordPress websites that have millions of orders um in the database and you need to move that website from point A to point B, one of the things that you might end up doing is um a search and replace in the database to change the URL. Okay. >> Very straightforward. Yeah. Um most people just use a plugin. Some people use WPCLI. Yeah. Um and you know some some people just use the tool right within their hosting platform, right? Um but what I've been working on uh again for about 18 months is I just keep taking the same work that I started after 01 Pro. It was ([View Highlight](https://read.readwise.io/read/01m3pspcky7zxmeem0gr19n5zw)) - Note: [H1] Accountable humans stay in the loop - Tags: atom/literature/h1 - [00:29:15] actually 01 Pro was the first one that was able to advance this idea for me. um that version of that code that I built in December of 2024, every new model release, I have it test that code and make it better and make it better and make it better. And about 3 months ago, I finally handed it off to my CTO and he then made it in order of magnitude better than what I had made because he had ideas around what we could do with with making it better, right? And and ultimately like I had gotten the code so good that it was like I I was convinced there was no no more room to improve it, right? And I had taken something that in a very large [00:29:59] database a search and replace could take 24 hours, literally 24 hours and you don't even know if it's working, right? Like you could be 23 hours and 59 minutes in and not know how far along you are, right? Not know that you're even that you're that close to the diamond in the mine, right? And we took this 24-hour task and um and I took it down to about an hour and then he took it down to 10 minutes and then I had an idea that took it to 10 seconds. And so now we've taken this work that's real like it's a real problem. You can't if you go to any host anywhere they'll go, "Oh I'll see you in 24 hours." That's how long that's going to take. ([View Highlight](https://read.readwise.io/read/01m3pspj44s3y2x9hggxr7h7f3)) - Note: [H3] Model iteration cut a 24-hour task to seconds - Tags: atom/literature/h3