Building The Web We Want With AI by Ronnie Burt 🔗 **Source:** [Building The Web We Want With AI by Ronnie Burt](https://www.youtube.com/watch?v=ZJ9rgAoWQk4) > [!note]+ Document Notes > tags: [AI, web-development, wordpress] > author: Ronnie Burt > source: All Things Open Conference > reference: https://www.youtube.com/watch?v=ZJ9rgAoWQk4 > date created: 2024-12-19 14:30 > > # 🏷️ Title: Building The Web We Want With AI > <!-- deep-read:sections start --> ## 🧰 Action Inbox - [H1] Implement observability and evals for AI tools — critical for monitoring AI performance 00:00 #ai-tools - [H2] Focus on human-in-the-loop AI rather than full automation — better outcomes 00:00 #product-strategy - [H3] Prepare for shift from traditional SEO to LLM-optimized content — traffic patterns changing 00:00 #seo ## 🧭 Table of Contents - [📝 Summary](#-summary) - [📖 Main Content](#-main-content) - [🗒️ Notes](#️-notes) - [📚 References](#-references) - [🗣️ Paraphrase](#️-paraphrase) - [📰 Newsletter](#-newsletter) - [✅ Action Items](#-action-items) - [🧪 Hypotheses](#-hypotheses) - [📊 Methodology](#-methodology) - [❓ Questions](#-questions) - [🔮 Future Research](#-future-research) - [💡 Implications](#-implications) - [🧠 Learning Styles Assessment](#-learning-styles-assessment) - [📊 Accessibility Assessment](#-accessibility-assessment) - [🌍 DEIB Assessment](#-deib-assessment) - [✅ Logical Validity](#-logical-validity) - [🛡️ DARVO Analysis](#-darvo-analysis) - [🎯 Conclusion](#-conclusion) ### 🔗 References - [WordPress.com AI Builder](https://wordpress.com/ai) - [DIA Browser](https://dia.so/) - [Langfuse Observability](https://langfuse.com/) ## 📝 Summary - **What happened**: Automattic's Ronnie Burt presented on how AI is reshaping web development and content creation, moving from hype to practical impact - **Why it matters**: Traditional web browsing and SEO are changing as AI-powered browsers and LLMs become primary ways users find information - **Key stakeholders**: Web developers, content creators, WordPress users, open source community - **Risks and tradeoffs**: Risk of losing traffic to AI summarization vs. opportunity for more targeted, authentic content - **Bottom line**: Focus on "human-in-the-loop" AI solutions rather than full automation, with emphasis on observability and evaluation systems ## 📖 Main Content Burt argues that we're at an inflection point where AI browsers like DIA, Comet, and Chrome with Gemini are changing how users discover websites. Instead of typing URLs, users ask questions and AI searches multiple sites simultaneously, summarizing results. This shift threatens traditional SEO while creating opportunities for more authentic, niche content. The presentation distinguishes between AI "hype" versus real "impact," advocating for human-centered AI tools rather than fully automated solutions. Key technical recommendations include implementing observability systems like Langfuse and evaluation frameworks to monitor AI performance before deploying new models. For WordPress specifically, Burt's team focuses on setting database options and theme configurations rather than generating code, leveraging the open source platform's existing structure that LLMs already understand from training data. ## 🗒️ Notes #### ⏱️ [00:00](https://www.youtube.com/watch?v=ZJ9rgAoWQk4&t=0) — [H1] Introduction and Context - Key idea: AI landscape changing faster than understanding can keep up - Evidence: "The more I learn, the more I think I know, the less I realize I know" #### ⏱️ [02:30](https://www.youtube.com/watch?v=ZJ9rgAoWQk4&t=150) — [H1] Are There Even Websites in the Future? - Key idea: New AI browsers changing how users discover content - Evidence: DIA browser has no URL bar, only asks questions instead #### ⏱️ [04:45](https://www.youtube.com/watch?v=ZJ9rgAoWQk4&t=285) — [H2] How AI Browsers Work - Key idea: AI browsers automate the "open 10 tabs" research process - Evidence: "AI is doing that same search, opening those 10 pages, going through them all" #### ⏱️ [06:15](https://www.youtube.com/watch?v=ZJ9rgAoWQk4&t=375) — [H1] SEO is Changing to AEO/GEO/AIO - Key idea: Traditional Google traffic declining, LLM traffic increasing - Evidence: "Decline in most traffic from Google directly, increase in traffic from LLMs" #### ⏱️ [08:30](https://www.youtube.com/watch?v=ZJ9rgAoWQk4&t=510) — [H2] What Still Works in New SEO - Key idea: Authority building and source citation remain important - Evidence: "LLMs love a good comparison" for product evaluations #### ⏱️ [10:00](https://www.youtube.com/watch?v=ZJ9rgAoWQk4&t=600) — [H2] Content Becoming More Structured - Key idea: Content needs clearer structure tied to schema.org - Evidence: Recipe sites will put recipes at top for LLM discovery #### ⏱️ [12:15](https://www.youtube.com/watch?v=ZJ9rgAoWQk4&t=735) — [H3] Hope for Less Bloated Websites - Key idea: SEO bloat practices may disappear as LLMs seek direct answers - Evidence: No more scrolling through recipe backstories to find ingredients #### ⏱️ [14:30](https://www.youtube.com/watch?v=ZJ9rgAoWQk4&t=870) — [H2] Local AI and Privacy Benefits - Key idea: On-device AI processing could improve privacy - Evidence: Apple Intelligence and Google local LLMs eliminating cookie tracking #### ⏱️ [16:45](https://www.youtube.com/watch?v=ZJ9rgAoWQk4&t=1005) — [H2] Need for Authenticity Standards - Key idea: Open source community should lead authenticity verification - Evidence: "This is where the open source community can have the most influence" #### ⏱️ [18:00](https://www.youtube.com/watch?v=ZJ9rgAoWQk4&t=1080) — [H1] Hype vs Impact Interactive Session - Key idea: Most AI applications are hype, impact comes with human involvement - Evidence: Audience participation revealing mixed reactions to AI trends #### ⏱️ [25:30](https://www.youtube.com/watch?v=ZJ9rgAoWQk4&t=1530) — [H1] Observability and Evals Are Critical - Key idea: Monitoring and testing AI outputs essential before deployment - Evidence: GPT-4o performed worse than 4.1 due to double-thinking in prompts #### ⏱️ [28:15](https://www.youtube.com/watch?v=ZJ9rgAoWQk4&t=1695) — [H2] WordPress AI Builder Approach - Key idea: Generate database options, not code, leveraging open source training - Evidence: "We're setting options in database, remembering design choices" #### ⏱️ [32:00](https://www.youtube.com/watch?v=ZJ9rgAoWQk4&t=1920) — [H1] Human-in-the-Loop is Key Theme - Key idea: AI impact comes from assisting humans, not replacing them - Evidence: "AI with the human, that's when we're going to get the impact" #### ⏱️ [34:30](https://www.youtube.com/watch?v=ZJ9rgAoWQk4&t=2070) — [H3] Observability Implementation Tips - Key idea: Use tools like Langfuse for tracing AI interactions - Evidence: Human categorization before LLM analysis produces better insights #### ⏱️ [38:45](https://www.youtube.com/watch?v=ZJ9rgAoWQk4&t=2325) — [H3] Evaluation Best Practices - Key idea: Keep evals simple pass/fail with explicit examples - Evidence: "Two or three examples improves eval accuracy significantly" ## 📚 References - [WordPress.com AI Builder](https://wordpress.com/ai) - [DIA Browser by Browser Company](https://dia.so/) - [Comet Browser by Perplexity](https://www.comet.com/) - [Langfuse Observability Platform](https://langfuse.com/) - [Schema.org Structured Data](https://schema.org/) ## 🗣️ Paraphrase - AI browsers are changing web discovery by answering questions directly instead of requiring users to visit multiple sites manually - Traditional SEO is evolving into AI-optimized content that emphasizes structure, authenticity, and recent information over keyword stuffing - The most impactful AI implementations keep humans in control while using AI as an assistant, particularly in areas requiring monitoring and evaluation ## 📰 Newsletter **AI is Reshaping Web Discovery - Here's How WordPress Sites Can Adapt** The way users find websites is fundamentally changing. Instead of typing URLs or clicking search results, AI browsers now answer questions directly by searching multiple sites simultaneously. For WordPress site owners, this shift from traditional SEO to "AEO" (AI Engine Optimization) means focusing on structured, authentic content rather than keyword games. The key opportunity? LLMs trained on open source code already understand WordPress architecture. Smart implementations generate database options and theme settings rather than custom code, making them more stable and maintainable. **Practical next steps:** Implement schema markup, put key information at the top of pages, and consider observability tools if adding AI features. Resources: [Learn WordPress](https://learn.wordpress.org/), [Developer Docs](https://developer.wordpress.org/), [WP-CLI](https://wp-cli.org/) ## ✅ Action Items #### WordPress - Core: Explore LLM-friendly content structure standards for improved discoverability - Documentation: Create guidelines for AI-optimized content architecture - Community: Develop best practices for human-in-the-loop AI implementations - Themes: Integrate schema.org markup as default practice - Plugins: Evaluate AI tools for observability and evaluation frameworks #### DevRel - Education: Create workshops on transitioning from SEO to AEO strategies - Developer experience: Document AI integration patterns that avoid code generation - Community management: Foster discussions on authenticity verification standards #### OSPO - Policy & Governance: Establish guidelines for AI tool evaluation and monitoring - Community Engagement: Support open source observability and evaluation tools - Strategic Alignment: Track AI impact metrics vs. hype indicators ## 🧪 Hypotheses - AI browsers will reduce direct website traffic while increasing qualified referrals from LLMs - Content structured with schema markup will perform better in AI search results than keyword-optimized content - Human-supervised AI implementations will outperform fully automated solutions in user satisfaction - Open source platforms like WordPress have advantages in AI integration due to public training data ### 📊 Methodology - A/B testing structured content vs. traditional SEO-optimized content for LLM referral traffic - User satisfaction surveys comparing human-in-the-loop vs. fully automated AI features - Performance analysis of schema-marked content in various AI search tools ### ❓ Questions - How will content creators maintain discoverability as AI browsers become primary discovery mechanisms? - What open standards will emerge for content authenticity verification? - How can smaller, niche websites compete with enterprise sites in AI-mediated search results? ### 🔮 Future Research - Development of open source authenticity verification protocols for web content - Analysis of local AI processing impact on website analytics and personalization - Study of user behavior changes in AI-first browsing environments ### 💡 Implications - **Practice**: Web developers need to prioritize structured data and observability over traditional SEO tactics - **Policy**: New standards needed for AI tool evaluation and human oversight requirements - **Security**: Local AI processing may reduce privacy risks but increases device-level security considerations - **Education**: Training programs must shift from traditional web development to AI-assisted development patterns - **Funding**: Investment should focus on observability and evaluation tools rather than fully automated AI solutions ### 🧠 Learning Styles Assessment Reading level estimate: 12th grade (technical presentation with specialized terminology) #### 👀 Visual learners - Present: Screenshots of AI browsers, comparison slides - Improve: Add diagrams showing AI search flow, visual framework comparisons #### 🔊 Aural learners - Present: Conference presentation format with speaker narration - Improve: Audio summary highlighting key technical distinctions #### 🗣️ Verbal learners - Present: Clear definitions of AEO, observability, human-in-the-loop concepts - Improve: More analogies comparing traditional vs. AI-powered web browsing #### 🧩 Kinesthetic learners - Present: Interactive hype vs. impact voting exercise, live Q&A - Improve: Hands-on workshop with Langfuse setup, eval writing exercises #### 🧠 Logical learners - Present: Clear cause-and-effect reasoning for AI browser impacts - Improve: Decision tree for choosing AI implementation approaches #### 🧑‍🤝‍🧑 Social learners - Present: Audience participation, conference networking opportunities - Improve: Discussion groups on authenticity standards, community tool development #### 🧘‍♀️ Solitary learners - Present: Individual reflection prompts on AI tool evaluation - Improve: Self-assessment checklist for AI readiness, personal experimentation guides #### 🌿 Naturalistic learners - Present: Ecosystem thinking about web platform evolution - Improve: Analogies to natural selection in AI tool adoption ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear heading hierarchy in presentation structure - Improve: Add slide descriptions for visual elements like browser screenshots #### Keyboard navigation - Present: Standard presentation controls available - Improve: Ensure demo tools mentioned are keyboard accessible #### Contrast and readability - Present: High contrast slides with clear typography - Improve: Verify all text meets 4.5:1 contrast ratio in materials #### Text resizing and reflow - Present: Scalable presentation format - Improve: Provide transcript in multiple formats #### Media and captions - Present: Video has complete transcript available - Improve: Add time-synced captions to video version #### Forms and interactions - Present: Simple Q&A interaction format - Improve: Multiple ways to submit questions during presentations ### 🌍 DEIB Assessment #### Cultural breadth - Present: Focus on global web standards and open source community - Improve: Include examples from different geographic regions, languages #### Inclusive language - Present: Technical terms defined clearly, avoids unnecessary jargon - Improve: More examples beyond English-language websites and tools #### Neurodiversity support - Present: Clear section breaks, consistent structure throughout - Improve: Provide content outline upfront, use consistent terminology #### Representation - Present: Diverse audience participation in Q&A section - Improve: Include varied perspectives on AI impact across different communities #### Psychological safety - Present: Acknowledges uncertainty, invites questions and disagreement - Improve: More explicit welcome for different viewpoints on AI adoption ### ✅ Logical Validity #### Strong reasoning - Deductive strength: Clear progression from AI browser behavior to SEO implications - Inductive strength: Well-supported generalizations from specific tool examples - Causal reasoning: Convincing links between AI adoption and traffic pattern changes - Analogical strength: Effective comparison between traditional and AI-mediated browsing #### Fallacies - False dilemma: Presentation of hype vs. impact as binary could oversimplify nuanced tools - Hasty generalization: Some conclusions about AI tool performance based on limited examples #### Tone of voice - Overall tone: Pragmatic and questioning, balancing skepticism with optimism about AI potential ### 🛡️ DARVO Analysis - Deny: No denial patterns detected - Attack: No attacks on accusers present - Reverse victim/offender: No victim/offender reversals identified - Emotional cues and rhetoric: Balanced, evidence-based presentation without manipulative language ## 🎯 Conclusion The web is transitioning from URL-based navigation to question-based AI discovery, requiring content creators to priorit <!-- deep-read:sections end --> ## 📺 Video ![](https://www.youtube.com/watch?v=ZJ9rgAoWQk4) --- # 📄 Full Transcript *Auto-generated from YouTube captions · timestamps roughly every 30 seconds · [watch on YouTube](https://www.youtube.com/watch?v=ZJ9rgAoWQk4)* **[00:05]** All right. Well, good afternoon. This is a lunch session, so I know a lot of you are balancing food on your laps. There are some tables up at the front if you didn't notice. You can come on up and join us. We'll keep this pretty informal for the lunch session. I know this is probably your 700th talk in the last two days on AI, so we'll try to keep it a little varied and a little bit different, too. My name is Ronnie Bert. That is my WordPress website that has links to some of my socials and some **[00:36]** more information about what I do and what I've done. And I work for a company called Automatic. Automatic is known for WordPress.com, for Woo Commerce, for Jetack, for Tumblr, and a whole bunch of other brands. And we really focus at our core on open source, open source community. Of course, WordPress is a large open source project in and of itself. And so I focus on our **[01:07]** AI teams on WordPress.com on the product side about how we can integrate AI in a useful way for our customers so that they can be more successful in achieving their goals. And this talk is a little bit different than the type of talk that I normally would give. I usually like tried to listen to my high school uh speech teachers. You know, we should have an arc that with a story that connects nicely. But the thing about AI and where we are right now or at least where I personally am is the more I learn. The more I think I know, the less **[01:39]** I realize I know. And I hope that a lot of you can come and join me in that space about figuring out what it is that we want to see with AI, how it's going to shape the web and the websites that people are building and all of the different tools around it. So the first question I want to start with is are there even websites in the future? And to be honest, this is I've been building websites in one way or another since like the mid to late 90s. This is **[02:11]** not a question that I thought I would be asking anytime soon. I've always been a huge evangelist for every small business, every dentist, every restaurant, every student in school, every person looking for a job, every nonprofit. We all need websites under our own domains um that we can control that we can move the content with us and not be part of some proprietary platform. But there are some obvious things changing that we need to pay pretty close attention to. And one of the **[02:44]** things that I've been playing a lot with lately are these new evolutions of web browsers, right? [snorts] And this is a screenshot of DIA. DIA is from the browser company recently acquired by Atlassian. Inside DIA is the homepage of Comet. Comet is another browser by Perplexity. There's also Chrome itself embedding Gemini features inside of Chrome more and more every single day. And I've been **[03:14]** spending a lot of time like how can we use these this new paradigm as we browse websites? And to be honest, I haven't found personally like a solution that like has really been a gamecher that's really changed the way that I interact with the web dayto-day. But I do see a seed there, something that is beginning to take place where we can just find what we're looking for faster, do what we're trying to achieve faster. And [snorts] just a note, since we're **[03:45]** here at all things open, all three of these are powered by the open source Chromium platform, which is just pretty cool in and of itself. But as I'm using these tools, one thing that really stood out to me in the DIA browser is that when you click on the new tab link, forever in browsers, we've been used to the next step. Put in a web address, put in a URL, a domain, a link. That's not the case in DIA. There's not **[04:16]** even a place to put my link like how do I what do I do? Where do I go? It wants you to ask a question. And then it asks uh the it does the search for you. [snorts] And so the way I'm thinking about it is that forever and ever, if you were like me, you search in Google, you would rightclick and open up 10 tabs of the first 10 results. And then you would go through them each one by one, hoping you find what you're looking for. And then if you remember, you'll close all those tabs when you're done. Well, that's what the AI is doing for you in these browsers is it's doing that same **[04:48]** search. It's opening up those 10 pages. It's going through them all and it's trying to find what you're looking for. Then it spits out a summarized result. This is changing the way people are getting to our websites. The top of the funnel, the way that we get people to the information that we're trying to get them to. And what that means bye-bye SEO after all this time is SEO really changing or is SEO something **[05:18]** different? What I'm seeing is there's definitely in almost every website that I kind of have access to a decline in most traffic from Google directly. The good news is there's an increase in traffic from the LLMs, from chat GPT, from Claude if sites are are lucky to have that. And there's a whole new alphabet soup going on and a fight with is this AEO, GEO, AIO. I don't know. I don't think **[05:50]** it's worth like really diving into all of these acronyms, but there are some subtle differences, but there are trends in all of these approaches, and it's basically the way that SEO is changing. So, the same things that have worked in SEO in the past, building authority, linking out to your sources or citing your sources, those are still great. That's still working with LLMs. And for whatever reason, they love a good comparison. So, you're seeing a lot more comparisons being written against **[06:20]** products or or whatever because that's how people are asking that question. you know, please compare for me these four hotels or these four apps that do whatever it is I'm trying to do. But what's different is the content itself or what we're seeing as is what people are building differently for their websites. Content's becoming a little clearer, a little more tied to schema, schema.org, or just some sort of predictable pattern in the content **[06:51]** that's being created. And that's leading us to a change, a real change in the way people are creating, editing, maintaining, building websites and the purpose that that website has. So for this little next phase of the presentation, these are just my personal like kind of what I'm seeing, what I'm feeling, what I'm hoping may happen. Um, but I'm hoping that there's more and more power in the most recent content that the LLMs are not their **[07:24]** source training data, but the ones that include search are pulling the most recent content. There's more and more of a need for our content to be unique, to have your own perspective, have your own like human voice in what it is that you're sharing. And this I'm kind of excited about is more websites that are more niche, more on a specific topic, more local to the community that you're in or the type of **[07:55]** well niche field that you're looking for. And if we're really lucky, this means way less bloat on websites and all of the SEO practices that have become so prevalent, right? [snorts] We all know you're looking for a recipe and you have to scroll and scroll and scroll to finally find the recipe at the bottom. Why? Because SEOs have decided this is a good way to game the system to, you know, pack in a whole bunch of keywords and interesting information at the top **[08:25]** so that they get the search traffic. But the LLMs, they're just looking for that recipe. So, we're going to see recipes at the top so that the LLMs find them faster. and we'll see the recipes using the recipe schema for exactly how like all the ingredients are listed in all of this. The other practice that we see a lot is uh content marketing and you'll write four, five, six blog posts all are basically the same with a slightly different keyword approach **[08:57]** and that's just, you know, not good for anyone really, right? What I'm hoping is that the focus will be on this source of truth on your website for that topic. And that's really well structured and really easy for humans to read, but also for the LLMs to find what they're looking for. This one's a little interesting and something I think we have to look forward to. You're seeing Apple **[09:28]** intelligence on the phone. We're seeing Google experiment with more local LLMs that can be ran in the browser. And I don't think it's very far-fetched to think that a lot of what now is currently being sent off to OpenAI or sent off to Gemini to be more local and on the edge. Again, if we're lucky, this means a great step forward for privacy because instead of the cookie tracking that we do now on all the websites to know who that person is and all the sites that **[09:59]** they're visiting, we'll be able to make inferences, send those inferences through the the local LLMs, and you won't be able to identify a specific person with that information. Other thing I hope we're seeing as we build the next version of the web is authenticity. And this is where I really think the open- source community can have the most influence because it's going to need to be some open standard. I've been looking and I haven't been **[10:30]** finding anything that like I think we're going to gravitate towards anytime in the really near future, but there are lots of people thinking about this. And this goes back to the idea of building authority on the website that you know that it's not just a bunch of LLM created content that the person that wrote the content is standing behind it and really validating what it is there. By the way, I was a middle school and **[11:01]** high school math teacher for a long time right out of college. I'm super used to being interrupted and like heckled or whatever. So like this can be informal. Well, it's a large room, but just shout out questions or something. And I'm going to try to get you a little more involved here in a minute. And then just this just an aspirational thing that as LLMs and just generative AI is creating more, maybe we will get more local, more niche, more human connections, maybe in person, and a lot **[11:32]** of this content uh overload. there's just a lot of digital waste being created. Basically, um we'll just find less value in over time. So, all this leads to kind of a phrase that I've kind of adopted. It's a little cheesy, but less hype, more impact. How do we get less hype from all the AI stuff and more impact with both within our businesses, within our products, for our end users, and the problems that they're trying to solve. **[12:03]** And so I want to play a little game. I've got a bunch of little uh just topics on the AI topic. And as I go to that slide, I want to see if you can just off the top of your head, your gut reaction. Is it hype or is there real impact here? And just kind of yell out what you're thinking. I kind of want to feel out what the crowd is thinking here. So we're going to try it. I'm going to flip the slide. I'm going to put up a word and then you just have to kind of yell hype or impact. **[12:39]** I think what I heard the most was Impact. [snorts] Right now, I'm living in hype for MCP. We've built it and uh for our products, we've made connectors. I'm playing with a bunch. There are a few that I find some real impact with. We do all of our product management in Linear. Linear has a really great MCP that that is useful, but MCP is just a connector for existing APIs in almost every case. It could have the same interaction could have been handled another way, maybe a more efficient way. [snorts] And I I'm just not sold yet **[13:11]** that our average user outside of the developer space is is really here. there's just too many unknowns uh around security, around authentication, around giving it permissions to write um and and all of these things. So, you know, if if this kind of surprises you or offends you, we can talk about it a little bit more later. Next word, content creation. And is AI having an impact on this or is this hype? **[13:45]** I I gonna believe that humans are going to figure out how to how to get past all of the fake created content that's out there and that we're going to through authenticity through just uniqueness. You can read when something is created by AI and know pretty much right now that it's created by AI. And [snorts] I have some thoughts on the next iteration of content creation that I'll get to later that will have I believe more impact but in and of its own just asking chatpt to create a blog post publishing that blog post. Some people are getting **[14:16]** some traffic and some success with that right now but long term I think that's all hype. AI is taking our jobs a lot of hype. [snorts] I agree. I think it's mostly hype. There's so much like this is also creating so many more jobs, right? And so many more opportunities. And the thing that I think it's doing with most jobs, especially in in the tech space, is that **[14:47]** I am a pretty terrible developer, but I've been, you know, hacking away, you know, the people that you hate that say they're dangerous. Well, now I'm really dangerous because I have cursor and I have cloud code and I can really build stuff that actually works and push it live. And I think that's only going to get better. I think all of us are kind of merging in many ways to be generalists and you'll still have your area of um expertise, but just the the way this gives us like when it's an **[15:17]** assistant and it can help improve your practice, it's I don't think that's hype. I think that's impact. Next word or words. AI note takers. This one I just threw in as a personal pet peeve. Uh if you have an AI noteaker in all your meetings, like I'm in a 100 meetings a day and sometimes there's more AI note takers in the meeting than humans. [snorts] So like if you're one of those people with one of those that auto adds to your **[15:48]** your meeting, your co-workers are probably just being kind by not telling you to please stop. Um, the real reason I think these are more hype is because who's reading those notes, right? Are we really going back and and using them and expecting them to be a good source of truth? There are some examples where this is more impact. I do use granola as not a pitch for granola, but the thing I like about it is before it spits out the notes, I give it my notes from the meeting. As the meeting is going on, I **[16:19]** type in the notes. I say, "These are the important pieces. This is what I want to remember. or this is my to-do. Then it incorporates that as the major piece and it fills out the rest of the notes around it. So, it's being that assistant there. It's not replacing. And that's a theme that I think we're going to see more and more. Next one. All the AI app builders. >> Hype. What's kind of weird because I work on an AI website builder and I don't Is it hype? Is it impact? I heard **[16:51]** a lot of hype. I tend to agree that most of them the way that they are currently working today is a whole lot of hype. And that's because if you actually use them, you go and you look at the code, you realize that a month later, six months later, dependencies, things are going to break. Who knows what kind of spaghetti nonsense is being built under the hood. That is going to get better and better. And then you have the other kind of end of the spectrum which is what we're doing with our WordPress AI builder which is we're not creating any **[17:22]** code in the building process. WordPress open source project. The beautiful thing about um WordPress and any open source project is that's what the LLMs have been trained on from the beginning because all that code's open and public in GitHub which is pretty good. And so we don't even have to give it a lot of context. you can go right into pick your favorite LLM and just say spit me out like markdown for the WordPress block editor for this page and it does it and it's pretty nice. Um but we're not writing code for that. We're setting **[17:56]** options in the database or we're remembering um you know different design choices that the user is asking for in in the theme. And so it's different than the app builder approach. And I I kind of think we're going to reach a middle like point over time. We're also experimenting with some code generation of plugins and blocks and things. And as they get a little safer, a little bit more stable, there'll be a time when you really need it to generate something for you so you can work with it. But having that foundation of the **[18:29]** platform that is open source and that you can build on top of, we have blocks, we have design language, all of the things that the LLMs can really use to build off of. That's better than a lot of the app builders that are out there. Now they are also creating their own design language or they're picking some some tools um to build from. But just as it is today, they're really great for prototyping, for experimenting, for coming up with **[19:00]** something quick, but not something you're probably putting in the enterprise anytime super soon. How we out of time? All right, next word. Well, two words. observability and evals. [snorts] These anybody anybody hype impact? Yeah, these are the buzzwords of just the last month or two. If you listen to all the podcasts, watch all the YouTubes um and for really good reason. So **[19:32]** if you take anything away from today, this is the area that I'm spending all of my time on pretty much within our product is figuring out with all of the interactions that our users are taking with AI tools, just the product in general. Observability is monitoring. So it's kind of analytics of what the AI is actually doing, what it's thinking. And evouse is then taking that and testing the output to see if it did what it said it was going to do or did **[20:03]** what you expected it to do. [snorts] And I guess I can tell a little story. Um, you know, what is it? A few months ago, GPT5 came out, right? And like any good product manager, you're like, "Oh, it's the next newest model. Let's just switch over." We were using 4.1 for our tools. It's newer. It's cheaper. It's faster. it's going to give better results. Well, we had evals in place and we knew immediately before pushing it live that the experience just degraded. The output **[20:35]** was worse. It was much slower. It was costing us a lot more. And it's kind of a somewhat of a funny story behind it. And we built it for 4.1. 4.1 is not a thinking, not a reasoning model. And so in all of our prompts, we were saying, "Hey, chatbt, remember to think here. Go and think about this." Well, then in five, it does that without you telling it to do that. So, it was thinking twice on every time, every interaction. It was thinking when we told it to think and **[21:06]** then it was thinking again because it's a reasoning model. And I don't really know the order if it which one was going first, but the output the result was from this newer, faster, better model that you know your your product manager says you should just switch to is worse. And we wouldn't have known that if we wouldn't have put in all of the the work in doing our having our evals in place and monitoring what folks are actually doing. So all this to say that this is the **[21:38]** theme of the day that AI in and of itself is a lot of hype. There is so much hype out there but AI with the human that's when we're going to get the impact and that's where we're seeing the impact. We have an annoying phrase human in the loop and that's what it is, right? And so that's everything from your notetakers in meetings to the developer developing product to the content creator creating blog posts or images or videos on Sora. **[22:09]** It's the human really that is still in control that really can put in their own personality and define what it is that they're trying to build. So I think I went through it really quickly. This is uh building the web we want which the whole point is we want it to be human centric first the humans building it not all the agents and that we have a long way to go to to really **[22:40]** get to the impact that we expect from this. You can play with our builder just to see what I'm talking about at wordpress.comai. But um you know this again was informal. I would love to just riff on any topics that piqu your interest if if you have any questions. Yeah. >> Yeah. So the question just so everybody **[23:13]** can hear is go back to the slide on observability and any any tips or any thoughts on how we can make sure that we're implementing this. So for us it's it's kind of an a product a lot of work you choose. I think I've seen several sponsors here that that have some tools for observability. We're using Langfuse at the moment and all of the interactions with the AI get fed into Langfuse as a as a trace and **[23:45]** the trace is the interaction the basically like a chat transcript of what the user inputs but where the value is we're telling the AI to share its thinking with us not with the user necessarily but in the trace like this is the tool I called this is the summary that I've made this is what I think I'm supposed to to this is what I've done and then it's a matter of going through them all and and then here's the the another thing that I I had a slide for actually and then I took **[24:15]** it out so I'll tell the little story. Um you know going along the theme of AI plus humans is where the real impact is. What I see all the time in teams I work with and just demos on YouTube or whatever is people think all right LLMs are really good at taking this data and summarizing it. So I'm just going to export from Langfuse all of these traces and I'm just going to throw it at LLM and it's going to tell me what their main pain points are and it's going to tell me what I should do next. Right? **[24:47]** And that's a good idea. I've fallen for that many times. But there is a way if you actually go through the lengthy process of getting much better value out of that LLM interaction when doing that. So I can describe that uh briefly or at least our practice is we export all of those and maybe it's a thousand traces or or whatever it is and the human who knows best goes through first and reads **[25:18]** as many as it takes in order to get a good idea of common trends. However you want to tag if you want to tag it or like this is the problem they're hitting or or whatever it is. And so the big question is, well, how many do I have to read? The best advice I've ever heard is you keep reading until you can't predict what the next outcome's going to be, until you're you're seeing the pattern. So maybe it's 50, maybe it's 100, maybe it's 500, depending on on what it is, but you are the one making the categories. You are the one saying, I I **[25:49]** see that in our case, just this uh this trace is a customer who's not getting the fonts that they want. This is a customer that's not getting the colors that they want. this is a customer not getting the copy or the content that they want. And then once I have those categories and I've defined the categories how it makes sense for my product, then I can upload the rest of them to the LLM and say, can you assign each of these traces to the closest category that matches that gives you a whole different uh level of just **[26:21]** validation behind the what you're doing and you know like you're going to get really good outcomes that way. So, I just this works for like user interviews, surveys, you know, anything [snorts] where you're trying to categorize or or look for trends. The human needs to find those trends first, predict what they are, put them through. Of course, you can run the whole thing once if it doesn't cost you too much and like get an idea of maybe some of them, but you'll find if you do that when you do it as a human, you're going to get different results. And uh so that's just **[26:51]** a some advice, I guess. Anything else? Yeah. >> Yeah. I mean, this is one that our our engineering teams are are really just starting to go back and implement like at a whole higher level. **[27:23]** Um the first builds is kind of the thing that gets skipped. It's kind of like unit testing or or something, you know, like it's nice, but uh we sometimes just kind of leave out of the process. I would say um first of all, we try to keep them super simple. It's a pass fail zero or one kind of result. I would also say that it does much better in the eval when you give it examples. This is something I didn't believe just because we're all making this up. I didn't realize that if I was **[27:54]** super explicit in the in the eval um you know this is an example of when the customer is not happy with the colors that they're getting. This is an example of when the customer is happy of the colors that they're getting. That just improves the the eval like output. It's much more accurate that way. If you give it two or three examples, it doesn't take a whole bunch. you don't want to like blow up your your context window. But that's that's one thing that we're trying to implement in the eval is just adding two **[28:24]** or three examples to each one to really help the LLM judge. And in um Langfuse that we use, there's even a thing called LLM as a judge, which is pretty handy to to write these. Yeah, it's so the question is for eval are they automated or are they manual? I it can be both. There's so we have automated evals for like just almost more like unit testing. Does things break? Do they work? Do they not? And **[28:55]** then there's manual going in and spot-checking and and training the it depends on the how complex the interactions are with the users as well if it can be automated or manual I would say. >> Yes sir. **[29:29]** the memory. Sorry. >> Oh, yes. >> Ethical memory. >> Yeah. Well, that you know, our company motto or one of our three goals is democratizing publishing. A lot of people use the democratizing thing, but that in the web we want, you have access to the tools to create what you want to create and you can host it anywhere. You **[30:00]** can take it with you with the AI and um ethical memory. >> Yeah, that is something that I I Yeah, that is something I worry about because the more context, the more stuff you throw at the LLMs doesn't necessarily mean the better output that it gives. And so I I think a lot of us have had this philosophy of like let's **[30:31]** not delete things from the web, let's keep it there. You have the, you know, the archives and that's not great for LLMs. If you have a 25-year-old website and you have some really outdated content, how do we give it the the information that um I mean, maybe you wanted to search the archives, but maybe you wanted to only show the most recent. I I don't I don't know that I have a great answer for you other than this is an important problem for us to to figure out. **[31:04]** >> Yes, sir. Yeah. **[31:35]** >> What do you see? >> [snorts] >> Yeah. So the question is around if I'm I can paraphrase um you know how can we get to one of the things that I talked about having more local more niche more authenticity around websites and empowering that creator to kind of own their content and own that space. And it's no fun building **[32:05]** something or creating content if no one's going to come read it and find it, right? And that's why those big platforms uh are so successful because that's where it sends you the eyeballs, sends you the traffic. And [snorts] that is something we've definitely got to figure out. I think this being a shift where people are going to start turning to their LLM of choice or their search tool, this is a new opportunity. You know, for a while, a lot of people still go to YouTube first for if they want a product review or if they want to, you **[32:37]** know, you're searching for a vacation and you kind of want to know like what that city looks like or or some reviews or something. We go to YouTube to search. Maybe that paradigm is going to change and it's going to be we start in the LLMs and [snorts] I don't I don't know how it's going to play out. I suspect there's going to be some of these um you know frontier models and these big AI companies that are going to be really paytoplay to be at the top of these search results and a whole another way **[33:07]** of ads that are not necessarily targeted but are more on the content or on the topics and I think there's going to then be room for other alternatives to to pop up to be this is the place that is more the open web and uh we're going to have an algorithm that's public and open source and transparent and it's going to just be the market's going to have to win out, right? Who ultimately I don't think most users care sadly. So, it's going to have to be who has the best user experience, who has the best end results and drives people **[33:37]** to what they're looking for. Um, that worries me a bit except maybe if it's paytoplay and you're always getting the same like Expedia result and I only dropped that because they're like one of the first launches in one of the I don't remember if it's Chat GPT or Claude recently on bringing their cards directly in. Um, maybe it's perplexity. I shouldn't spread rumors, but um you know that's it's a good question, but I think we need to figure out the **[34:09]** it's the top of the funnel. It's driving eyeballs to those those pages. That's the trick. And yeah, we've got to figure that out. [snorts] **[34:40]** Yeah. >> Yeah. It's a good question. I don't know that we're ever going to have a great way of identifying the fraud or that completely AI generated stuff, but I'm thinking the alternative is going to be we're going to have trusted verified sources for created like human created or at least human assisted um highly assisted content and kind of flip it and go the other way and through um I mean it's **[35:12]** kind of the same way that it's been you we'll probably all have different sources that we trust and don't trust and that's just it's going to be an interesting I I'm not too worried about there's going to be lots of examples of deep fakes and things that are just created that are totally false that people are going to fall for and fraud I am worried about that but I think we're going to it's just going to be so much noise that **[35:44]** we'll figure out what rises to the top somehow. I don't know. That's it's wishful thinking maybe. Yes. **[36:19]** I think if I I'm having a hard time hearing, but I know you asked about cookies and that that's being used less and so when search happens through the LLMs, how does that translate to like how we're Yeah. I mean, I think personalization of **[36:49]** content is both like a blessing and a curse of of we should be able to write content on the fly on a website that's like tailored to it knows that I like tennis and I don't know what else. I live in Texas and the examples written on a sales page could be written directly to me. >> [snorts] >> The challenge with that will be how do LLMs like save that content to like send you to the top level of the interaction. I think we'll see more of those **[37:20]** personalized experiences like outside of sales landing pages or marketing pages and more like inside the product um once you're kind of in and then they'll have you'll probably be logged in and then they'll be able to track you and fingerprint you anyway probably at least in the short term. That's kind of how I see it. All right. Well, this has been fun. Thanks for having lunch with us and uh I will be standing outside at the automatic booth right outside these doors for the afternoon if if you want **[37:51]** to say hello. Thanks.