Good Bots, Bad Bots How to Win With Bot Management

Good Bots, Bad Bots: How to Win With Bot Management ![rw-book-cover](https://img.youtube.com/vi/P-0kzkV-Z48/maxresdefault.jpg) 🔗 **Source:** [Good Bots, Bad Bots: How to Win With Bot Management](https://www.youtube.com/watch?v=P-0kzkV-Z48) > [!note]+ Document Notes > tags: #bot-management #web-security #ai-crawlers #cloudflare #web-traffic #cybersecurity > author: WP Engine > source: YouTube > reference: https://www.youtube.com/watch?v=P-0kzkV-Z48 > date created: 2025-01-11 10:30 > > # 🏷️ Title: Good Bots, Bad Bots: How to Win with Bot Management >  ## 🧰 Action Inbox - [H1] Implement visibility tools to understand current bot traffic patterns — 00:00 #web-security - [H2] Set up logging rules before blocking to test mitigation impact — 15:30 #bot-management - [H3] Define bot strategy based on site persona and industry context — 25:45 #strategy ## 🧭 Table of Contents - [[#📝 Summary]] - [[#📖 Main Content]] - [[#🗒️ Notes]] - [[#📚 References]] - [[#🗣️ Paraphrase]] - [[#📰 Newsletter]] - [[#✅ Action Items]] - [[#🧪 Hypotheses]] - [[#📊 Methodology]] - [[#❓ Questions]] - [[#🔮 Future Research]] - [[#💡 Implications]] - [[#🧠 Learning Styles Assessment]] - [[#📊 Accessibility Assessment]] - [[#🌍 DEIB Assessment]] - [[#✅ Logical Validity]] - [[#🛡️ DARVO Analysis]] - [[#🎯 Conclusion]] ### 🔗 References - [WP Engine Global Edge Security](https://wpengine.com) - [Cloudflare Bot Management](https://cloudflare.com) ## 📝 Summary - Bot traffic now comprises over 50% of internet traffic, creating new challenges for website owners - Traditional good/bad bot classification is insufficient; a "gray area" of ambiguous bots requires nuanced management - Visibility and gradual implementation of bot management strategies are essential for avoiding blocking legitimate users - Industry collaboration and tool integration are key to developing effective bot management policies - WP Engine has mitigated over 75 billion malicious requests in 2025 through partnership with Cloudflare ## 📖 Main Content The panel discussion reveals how dramatically web traffic composition has changed, with sophisticated AI crawlers and automated systems fundamentally altering how we define legitimate versus problematic traffic. The experts explain that while traditional "bad" bots (clearly malicious) and "good" bots (like Google crawlers) are relatively easy to categorize, a growing middle layer of "gray bots" presents complex decisions for site owners. Brian Becker from Cloudflare emphasizes that the internet was originally designed for human browsing, but the current reality shows bot traffic dominating. Crystal Okconor from WP Engine shares concrete metrics about their platform's bot mitigation efforts, while Peter John Hunt from Useful Group provides agency perspective on how this affects day-to-day client management. The discussion emphasizes practical approaches: gaining visibility first, implementing cautious testing through logging rather than immediate blocking, and understanding that bot strategy must be customized based on site purpose, industry, and business goals. ## 🗒️ Notes #### ⏱️ [00:00](https://youtu.be/P-0kzkV-Z48?t=0) — [H1] Introduction and Expert Panel - Key idea: Panel includes experts from Cloudflare, Useful Group, and WP Engine discussing bot management - Evidence: "Today we've got some really amazing experts from the community all to talk about the hot topic bots" #### ⏱️ [02:15](https://youtu.be/P-0kzkV-Z48?t=135) — [H2] Traffic Composition Has Fundamentally Changed - Key idea: Over 50% of internet traffic is now bots, not humans - Evidence: "Today over 50% of the traffic on the internet is all bots" #### ⏱️ [03:30](https://youtu.be/P-0kzkV-Z48?t=210) — [H3] The Gray Bot Problem - Key idea: Most challenging category is neither clearly good nor bad bots - Evidence: "There's this middle layer. I like to call it the gray bots... maybe some site personas want that. Some don't" #### ⏱️ [05:45](https://youtu.be/P-0kzkV-Z48?t=345) — [H2] Five Years of Change in Six Months - Key idea: AI acceleration has compressed years of gradual traffic evolution into months - Evidence: "In a lot of ways, it feels like five years of traffic has all been condensed into the last six months" #### ⏱️ [08:20](https://youtu.be/P-0kzkV-Z48?t=500) — [H1] Preventing Human User Blocking - Key idea: Challenge-based mitigation is safer than hard blocking for ambiguous traffic - Evidence: "A bot mitigation solution with a capture is obviously going to be better than a solution that just hard blocks" #### ⏱️ [11:10](https://youtu.be/P-0kzkV-Z48?t=670) — [H2] Incentivizing Good Bot Behavior - Key idea: Providing proper pathways encourages bot operators to self-declare and follow rules - Evidence: "If you offer a mechanism for that operator to get this same information from you without scraping... that incentivizes them to start self-declaring" #### ⏱️ [13:45](https://youtu.be/P-0kzkV-Z48?t=825) — [H1] Visibility Is the Essential First Step - Key idea: You must see and categorize traffic before making management decisions - Evidence: "You really have to gain visibility. Step one, you have to be able to see the kinds of traffic you're receiving" #### ⏱️ [15:30](https://youtu.be/P-0kzkV-Z48?t=930) — [H2] Test Before You Block Strategy - Key idea: Log potential blocking rules to understand impact before implementing - Evidence: "You can log the traffic very specifically with a rule and see well how much and what kind of traffic is this rule actually going to mitigate" #### ⏱️ [18:15](https://youtu.be/P-0kzkV-Z48?t=1095) — [H3] Rate Limiting as Middle Ground - Key idea: Gradual traffic control through rate limits instead of immediate blocking - Evidence: "You could also do things like rate limiting, which still allows the traffic in. It just kind of slows it down a little bit" #### ⏱️ [19:45](https://youtu.be/P-0kzkV-Z48?t=1185) — [H1] WP Engine's Massive Bot Mitigation Scale - Key idea: WP Engine has blocked over 75 billion malicious requests in 2025 - Evidence: "WP Engine has mitigated over 75 billion malicious requests just in 2025 with Cloudflare's help" #### ⏱️ [21:20](https://youtu.be/P-0kzkV-Z48?t=1280) — [H2] New Traffic Visibility Features - Key idea: WP Engine released traffic categorization and geographic insights in user portal - Evidence: "We've just recently released an open beta for traffic and visibility in our user portal" #### ⏱️ [23:10](https://youtu.be/P-0kzkV-Z48?t=1390) — [H3] Edge Caching for Bot Protection - Key idea: Full page caching at edge reduces server load from both good and bad bots - Evidence: "Edge full page cache... all that traffic isn't coming directly to where your site is hosted onto the server" #### ⏱️ [25:45](https://youtu.be/P-0kzkV-Z48?t=1545) — [H2] Site-Specific Strategy Required - Key idea: Bot management strategy must be customized based on client industry and intent - Evidence: "You now got to set your strategy of what am I doing with bots directly related to who your client is what's their industry what is their intent" #### ⏱️ [28:30](https://youtu.be/P-0kzkV-Z48?t=1710) — [H1] Take a Breath - Community Approach - Key idea: The entire internet community is learning bot management together - Evidence: "Take a breath because the entire internet community is dealing with this at the same time and we're all dealing with it together" #### ⏱️ [30:15](https://youtu.be/P-0kzkV-Z48?t=1815) — [H2] Tool Recommendations for Getting Started - Key idea: Start with Cloudflare for visibility, integrate with performance monitoring tools like New Relic - Evidence: "For us, we start with something like Cloudflare... then what's helpful to us is... understanding beyond even the Cloudflare layer" ## 📚 References - [Cloudflare Bot Management Solutions](https://www.cloudflare.com/application-services/products/bot-management/) - [WP Engine Global Edge Security](https://wpengine.com/platform/global-edge-security/) - [New Relic Performance Monitoring](https://newrelic.com/) ## 🗣️ Paraphrase - Bot traffic has exploded from a minor concern to over half of all internet traffic, creating unprecedented management challenges for website owners - The traditional binary of good versus bad bots is insufficient for today's landscape, which includes a large "gray area" of ambiguous automated traffic - Successful bot management requires gaining visibility first, testing policies carefully through logging, and customizing approaches based on specific business needs and industry context ## 📰 Newsletter **Bot Traffic Boom: How WordPress Sites Can Stay Ahead** The web has fundamentally changed. Over 50% of traffic is now automated, and that includes everything from helpful AI assistants to malicious scrapers targeting your WordPress site. **Key Takeaway**: Don't panic-block traffic. Start with visibility tools to understand what's actually hitting your site, then implement gradual controls like rate limiting and managed challenges. **Practical Win**: WP Engine customers can now access detailed bot traffic analytics in their user portal, showing traffic by category, verification status, and geographic origin. This visibility is your first step toward smart bot management. **What's Next**: Edge caching is rolling out more broadly, which helps by serving content to both good bots and humans from the edge without hitting your server. Resources: [Learn WordPress Security](https://learn.wordpress.org/), [WordPress Developer Handbook](https://developer.wordpress.org/), [WP-CLI Security Commands](https://wp-cli.org/) ## ✅ Action Items #### WordPress - Core: Integrate bot detection APIs into core security functions - Security: Develop bot management guidance for WordPress.org documentation - Hosting: Create standardized bot visibility dashboards across hosting providers - Plugins: Review security plugin bot detection capabilities and accuracy - Community: Share bot management best practices through meetups and WordCamps #### OSPO - Policy & Governance: Establish bot traffic policies for open source projects - Security Management: Include bot-related vulnerabilities in security scanning - Community Engagement: Fund bot management tooling for WordPress ecosystem #### DevRel - Education: Create workshops on bot traffic analysis and mitigation - Developer experience: Document bot management APIs and integration patterns - Community management: Facilitate discussions between hosting providers on bot strategies ## 🧪 Hypotheses - Sites with proper bot management see reduced server costs and improved human user experience - Edge caching significantly reduces the impact of both good and malicious bot traffic on origin servers - Industry-specific bot management strategies are more effective than generic approaches - Gradual implementation of bot controls results in fewer false positives than immediate blocking ### 📊 Methodology - Analyze server performance metrics before and after bot management implementation - Track user experience metrics during bot mitigation rollouts - Compare industry-specific versus generic bot management rule effectiveness - Measure false positive rates between different mitigation approaches ### ❓ Questions - How can WordPress core better support bot management without impacting site performance? - What standardized bot management APIs should hosting providers implement? - How do we balance legitimate AI training needs with site owner preferences? ### 🔮 Future Research - Develop WordPress-specific bot behavior detection patterns - Create industry-specific bot management templates for different site types - Investigate automated bot policy negotiation protocols ### 💡 Implications - Practice: Website owners need new skills in traffic analysis and bot management - Policy: Need industry standards for bot behavior and identification - Security: Traditional security models insufficient for current bot landscape - Education: WordPress community needs comprehensive bot management training ### 🧠 Learning Styles Assessment Reading level estimate: 12th grade #### 👀 Visual learners - Present: Traffic charts and bot categorization diagrams mentioned - Improve: Visual flowcharts for bot detection decision trees #### 🔊 Aural learners - Present: Panel discussion format with expert dialogue - Improve: Downloadable audio summary with key technical terms defined #### 🗣️ Verbal learners - Present: Clear explanations of technical concepts and analogies - Improve: Glossary of bot management terminology #### 🧩 Kinesthetic learners - Present: Hands-on tool recommendations (Cloudflare, New Relic) - Improve: Step-by-step implementation guides #### 🧠 Logical learners - Present: Decision frameworks for bot categorization - Improve: Flowcharts for bot management rule implementation #### 🧑‍🤝‍🧑 Social learners - Present: Panel discussion format encouraging community collaboration - Improve: Facilitated discussion groups for bot management strategies #### 🧘‍♀️ Solitary learners - Present: Technical analysis and individual tool exploration - Improve: Self-assessment worksheets for bot traffic analysis #### 🌿 Naturalistic learners - Present: Ecosystem thinking about internet traffic patterns - Improve: Analogies to natural systems for understanding bot behavior ### 📊 Accessibility Assessment #### Screen reader semantics - Present: Clear speaker identification and topic transitions - Improve: Structured headings and speaker role descriptions #### Keyboard navigation - Present: Standard video controls available - Improve: Timestamped navigation links #### Contrast and readability - Present: High contrast video interface - Improve: Transcript formatting with proper heading hierarchy #### Text resizing and reflow - Present: Responsive video player - Improve: Resizable transcript text #### Media and captions - Present: Full transcript provided - Improve: Professional closed captions and audio descriptions #### Forms and interactions - Present: Basic video controls - Improve: Accessible chapter navigation ### 🌍 DEIB Assessment #### Cultural breadth - Present: Global perspective on internet traffic challenges - Improve: Examples from different geographic regions and markets #### Inclusive language - Present: Technical terms explained clearly - Improve: Avoid assuming prior security knowledge #### Neurodiversity support - Present: Clear structure with defined sections - Improve: Visual cues for topic transitions #### Representation - Present: Mix of company perspectives and roles - Improve: Include diverse voices from different organizational sizes #### Psychological safety - Present: Collaborative, learning-focused tone - Improve: Emphasize that everyone is learning this together ### ✅ Logical Validity #### Strong reasoning - Deductive strength: Clear progression from traffic changes to management needs - Inductive strength: Examples from multiple industry experts support conclusions - Causal reasoning: Direct connection between bot traffic and site performance issues - Analogical strength: Effective comparisons between past and present traffic patterns #### Fallacies - Appeal to numbers: "75 billion requests mitigated" used for credibility without context of total traffic - Hasty generalization: "Take a breath" advice may oversimplify urgent security situations #### Tone of voice - Overall tone: Collaborative and educational, emphasizing shared learning and community problem-solving ### 🛡️ DARVO Analysis - Deny: No denial patterns present - Attack: No attacking language toward any stakeholders - Reverse victim/offender: No victim/offender reversal present - Emotional cues and rhetoric: Reassuring, collaborative tone focused on shared challenges ## 🎯 Conclusion The panel makes clear that bot management is no longer optional for serious web properties. With over 50% of internet traffic now automated, website owners must develop sophisticated strategies for distinguishing between helpful, harmful, and ambiguous bot traffic. The practical next steps are clear: gain visibility into current traffic patterns, implement cautious testing approaches through logging before blocking, and customize policies based on specific business needs. The emphasis on community collaboration and shared learning reflects the reality that this challenge is affecting everyone simultaneously. For WordPress site owners, the integration of bot management tools with hosting platforms like WP Engine's new traffic visibility features represents a crucial evolution in making these capabilities accessible without requiring deep technical expertise. ## 🧱 Refactor --- ## Full Transcript [00:22] Hello everyone. Welcome to today's [00:23] session, good bots, bad bots, how to win [00:26] at bot management. I am Jenna Menu, a [00:28] senior product marketing manager here at [00:30] WP Engine and your host for this panel. [00:32] Today we've got some really amazing [00:34] experts from the community all to talk [00:36] about the hot topic bots. I'd like to [00:38] start by introducing the group here. We [00:40] have Brian Becker who is a director of [00:43] product at Cloudflare really focused on [00:45] web integrity and trust. We have Peter [00:47] John Hunt who is the chief technology [00:49] officer at useful group and Crystal [00:51] Okconor who is a senior product manager [00:54] here at WP Engine focused on all things [00:56] platform and network edge. We have [00:59] decades of experience at the [01:01] intersection of both the technical [01:02] endpoint and infrastructure all the way [01:04] through to customer impact here on the [01:06] call. So let's dive in. Um first [01:09] question for the group. Uh now we know [01:12] today's site traffic looks completely [01:14] different than it did six months or even [01:16] a year ago. How has the rise of [01:18] sophisticated AI crawlers changed the [01:20] way that we defined good traffic, bad [01:23] traffic and good bad bots? If we look at [01:27] the trajectory of the internet, the [01:30] internet was really created for humans. [01:32] Humans browsing websites. Eventually we [01:35] bring in the Google of the world and you [01:37] have search engines and search engines [01:39] bring in crawlers and the humans of the [01:42] world decided oh search engine crawlers [01:45] are probably okay because we want to get [01:47] found today over 50% of the traffic on [01:50] the internet is all bots [01:53] and it's not just search engine crawlers [01:56] it's agentic AI it's training it's um [02:00] looking for uh AEO and GEO and security [02:06] and all different kinds of things that [02:09] these bots are doing. Some of it is [02:12] good. Some of it, the majority of site [02:15] owners out there recognize that's [02:16] something that we want. Some of it is [02:19] downright malicious. And we can look at [02:22] that and say, okay, there's pretty much [02:24] nobody out there who actually wants [02:27] that, right? There's this middle layer. [02:30] I like to call it the gray bots. And [02:32] that graybot area is the stuff that well [02:37] maybe some site personas want that. Some [02:40] don't. Some aren't sure. Some want it, [02:43] but they want it in limited amounts. And [02:46] it's kind of this gray area and how you [02:48] manage not just good and not just bad, [02:52] but what do you do with this kind of [02:54] gray messy middle? And that's a [02:57] different place than we've been in the [02:59] last six months, a year, five years ago. [03:03] >> Yeah, I I would uh definitely agree with [03:06] that. In the past, it was much easier to [03:09] define what is bad for the majority of [03:13] the world. And if you get rid of that, [03:14] you're left with mostly good and maybe a [03:17] little bit of gray area to deal with. [03:18] And today, um, the bad things are all [03:22] still there, but that there's way more [03:23] in the gray area to use your your [03:25] definition. And like [03:28] getting a handle on that, understanding [03:30] what's going on. Um even understanding [03:33] like what these bots are because like [03:35] your average person is not going to know [03:38] uh the catalog of thousands of bots from [03:39] the world is is probably like the first [03:42] step everyone has to start getting ready [03:44] to take or start taking now is probably [03:47] a better way to say it. [03:49] >> Yeah. In a lot of ways, it feels like um [03:51] five years of traffic has all been [03:53] condensed into the last six months. And [03:55] the patterns that were there that [03:57] everybody could kind of take for granted [03:58] that we didn't even know we were taking [04:00] for granted of the fact that traffic was [04:03] predominantly human allowed us to [04:05] actually have a lot of things that were [04:06] just assumptions that we didn't have to [04:07] worry about. And so even from um you [04:10] know our our perspective as an agency uh [04:12] it's been everybody's reeling. You know, [04:15] for a long time clients don't want to [04:16] know and worry about traffic. They just [04:18] want to know that they're getting good [04:20] traffic. And a lot of that was kind of [04:23] built into Google's ecosystem. And so [04:25] there was there was these patterns that [04:26] we didn't know were going on that [04:29] allowed a lot to not have to be worried [04:30] about. Now everybody has to worry about [04:32] a lot because you want the good traffic, [04:34] but there's a lot more traffic to deal [04:36] with. And obviously even from a customer [04:38] perspective, the moment you've got your [04:40] site down or you don't know, you know, [04:43] and and the patterns of what traffic is [04:45] what and distinguishing that has has [04:46] just gotten way more complicated over [04:48] the past even 18 months to where I think [04:51] everybody's reeling a little bit from, [04:53] you know, what is the new world that [04:54] we're in now. [04:55] >> One thing I think uh several of you [04:57] mentioned is just making sure that you [05:00] have the path for that good traffic or [05:02] that human traffic still available on [05:04] the site. How do you make sure that real [05:06] visitors aren't accidentally blocked or [05:08] frustrated if they might fall into that [05:10] unknown category? [05:11] >> It's a complex answer. First, um, [05:17] you know, if you carve out the path for [05:20] the good bots, uh, and you carve out the [05:22] path to block the bad bots. And so, [05:24] we're just left talking with the gray [05:25] area. Like you said, uh, there's a lot [05:27] of areas ways you can start on this [05:30] problem. like first uh a bot mitigation [05:32] solution with a capture is obviously [05:35] going to be better than a solution that [05:37] just hard blocks. Um I you still ideally [05:41] don't want to show captions to good [05:42] users because that's friction, but it's [05:44] still better than hard blocking them. [05:47] Um, you can also look at the patterns of [05:51] activity you see and like what you [05:52] expect from your site and kind of [05:54] customize uh rules in your solution or [05:57] or in your own uh homegrown setup to you [06:00] know if you don't ever expect traffic [06:02] from certain areas [06:04] um don't allow it. You can work with [06:06] teams that [06:07] if you have any marketing events or like [06:10] promotion events and you know there's [06:11] going to be a huge flood of traffic [06:13] during a specific time, you might want [06:15] to work with uh your security team or or [06:18] if if that's you yourself to like loosen [06:21] up restrictions during that period to [06:23] just reduce the consequences of [06:27] potentially blocking humans. Um, [06:31] and finally, I think [06:34] going back to giving a good path to the [06:37] good bots, that actually makes the gray [06:39] area problem slightly easier because [06:41] that incentivizes [06:43] bot operators to like lift themselves up [06:46] out of this gray area. Um, an example [06:48] would be, uh, hypothetical example would [06:51] be a bot that's scraping you, um, but [06:54] with good intentions and like perhaps [06:56] they're like trying to like do this as [06:58] part of an aggregator that's going to [07:00] result in traffic being driven to your [07:01] site. And if you knew about this, you [07:04] would potentially like decide that you [07:06] want it. It's just right now you don't [07:08] know about it. If you offer a mechanism [07:10] for that operator to like get this same [07:12] information from you without scraping [07:14] you, um that incentivizes them to start [07:18] self-declaring and playing by your your [07:20] rules. And now you've like pulled some [07:21] of the problem space out of the gray [07:23] area and into the like well understood [07:26] area that makes it easier then to like [07:29] detect the remaining bots in humans like [07:32] indirectly resulting in less blocks to [07:34] humans. I think you really have to gain [07:37] visibility. [07:39] Step one, you have to be able to see the [07:41] kinds of traffic you're receiving. And [07:43] historically speaking, where did we all [07:45] go? We went to Google Analytics, right? [07:47] But Google Analytics is doing some [07:49] filtering on their own to show you what [07:51] they think you want to see on some level [07:53] rather than giving you a full picture of [07:55] the total traffic that's actually [07:57] hitting you. You need to be able to see [07:59] that traffic whatever tool you're using [08:01] whether it's through uh the visibility [08:03] that WP Engine provides if you are a WP [08:06] Engine customer or whatever web host [08:08] you're using or whether you're going [08:09] directly through something like a [08:11] cloudflare uh that offers you all kinds [08:14] of information through raw logs or [08:16] through their user portal or however you [08:18] choose to access you have to get [08:19] visibility and be able to bucket that [08:22] stuff. Um, but one of the other things [08:24] that's really useful and you're talking [08:26] about, hey, this is traffic that I I'm [08:28] not sure that I really want it. I don't [08:30] think I want it or I think it's [08:31] negatively impacting my site is if [08:34] you're able through your CDN or through [08:36] your firewall, this is something we do [08:39] with Cloudflare very often. there's [08:41] something you think that you want to [08:42] mitigate on some level, you can log the [08:46] traffic very specifically with a rule [08:48] and see well how much and what kind of [08:51] traffic is this rule actually going to [08:53] mitigate. You're not taking any action. [08:56] You're watching to see what is going to [08:58] happen and that allows you to take a [09:00] very cautious approach. And then u you [09:03] know Brian spoke to a managed challenge [09:06] that's really great because if it [09:07] detects that you're a human you just go [09:09] right through. if it's unsure or it's [09:12] positive to hear a bot, it's going to [09:15] it's going to mitigate that traffic. You [09:17] could also do things like rate limiting, [09:19] which still allows the traffic in. It [09:21] just kind of slows it down a little bit. [09:24] So, you don't have to go straight to a [09:26] block. You can start by watching by [09:28] gaining some visibility by saying, "If I [09:31] took action, what would it actually do?" [09:34] And I think that really helps taking [09:38] that cautious approach so that you're [09:40] not blocking the humans that you want [09:43] and learning how to take action against [09:46] what you don't want. [09:49] >> I like [09:51] what you said about visibility. I [09:53] completely agree. That's the most [09:55] important step and the first step. And [09:57] like setting up a rule and putting it to [10:00] log is a a great way to approach the [10:02] problem. Well, what what you also get [10:04] from that is, you know, not just like [10:06] what will this block and will I [10:08] accidentally hurt my real users, but [10:11] that kind of gives you information, you [10:13] know, what is this traffic doing? Like [10:16] at the end of the day, when there's a [10:18] lot of reasons why we want to block [10:20] bots, but I think you can bucket most of [10:22] those reasons into like it somehow [10:24] damages your site or it results in fraud [10:27] and abuse. Um, and if somehow you could [10:30] prevent those two things from happening, [10:32] it's it's maybe a kind of a different [10:34] question to ask. If these bots, going [10:36] back to agents are, you know, buying [10:39] things and they're good purchases, uh, [10:41] logging this behavior could help you [10:43] determine that and make a decision like [10:45] maybe you don't want to block it or like [10:47] reach out and contact the bot director [10:50] uh, directly and see if you can come [10:52] together with some kind of deal. [10:54] Crystal, I know um you've been working [10:56] on some really exciting things specific [10:58] to WP Engine customers. Do you want to [11:00] share a little bit of what's been on [11:02] your desk? [11:02] >> If you didn't know, WP Engine has [11:05] mitigated over 75 [11:08] billion malicious requests just in 2025 [11:12] with Cloudflare's help. That's a huge [11:15] number. That's that's pretty amazing. Um [11:18] I think the team has done a fabulous [11:20] job. We've just recently released an [11:23] open beta for traffic and visibility in [11:25] our user portal. And this gives you [11:27] access to see things like traffic by [11:30] agent category. So you can see, oh, this [11:33] is categorized as security or oh this is [11:35] categorized as SEO, which is really [11:38] helpful insight. And we also provide [11:41] information on is this a bot? Is it [11:44] verified? Is it unverified? [11:47] Um and what region, what country is it [11:50] coming from? So if you know that you [11:52] want to expect a higher percentage of [11:54] human visitors from a specific region, [11:57] we now provide that information in the [11:58] user portal. So if you haven't checked [12:00] it out yet, it's live today. It's [12:02] available for our Global Edge security [12:04] customers and it's pretty neat. Coming [12:07] soon, especially for our global edge [12:09] security customers, is the ability to [12:12] actually implement some level of control [12:15] over certain bot categories and certain [12:17] bot regions. So, you would be able to, [12:20] for example, manage challenge straight [12:23] out of a specific region that you know [12:27] that maybe that's not traffic that you [12:29] think you actually need to be receiving. [12:32] as there's a lot of really cool things [12:34] happening on the bot front at WP Engine. [12:37] Another thing that's upcoming towards [12:39] the end of this month is we're going to [12:41] be rolling out edge full page cache more [12:44] broadly across our advanced network and [12:45] our global edge security network. And [12:48] this brings caching, full page HTML [12:52] caching at the edge. Uh, which is good [12:55] for humans because they'll get your [12:58] content faster. And it's also good for [13:01] the bots problem because all that [13:03] traffic isn't coming directly to where [13:06] your site is hosted onto the server, [13:09] which means you see increased [13:11] performance on your site and that's only [13:13] good things. So, all kinds of cool stuff [13:16] coming at WP engines. Keep your eyes [13:18] peeled. [13:19] >> Now, Crystal and Brian have both been [13:21] involved in um building some really [13:23] great infrastructure and and applicable [13:25] tooling in that area. I do want to take [13:27] us back a little bit more high level for [13:29] a moment. Peter John um thinking as an [13:32] agency that just started to look at this [13:35] problem, approach this problem. Um, [13:38] obviously you and your agency uh have [13:40] been, you know, diagnosing and and [13:43] trying to figure out which strategy [13:44] works for your clients. Um, what would [13:48] you do now in being as far as you are as [13:51] a as a good first step to identify, you [13:54] know, what comes first? What do we what [13:56] do we do first when we know we have this [13:58] problem? [13:58] >> There there's a couple categories top of [14:00] mind. Uh first is um you know from a [14:04] visualization perspective part of the [14:05] reason that's most important for us is [14:07] because you now got to set your strategy [14:10] of what am I doing with bots uh directly [14:12] related to who your client is what's [14:14] their industry what is their intent [14:15] right there used to be the broader [14:17] patterns of okay most sites don't need [14:19] XML RPC to be uh allowed to be hit right [14:23] now it's much more of okay I I got to [14:25] know not just the specific traffic but [14:27] the specific traffic in the context of [14:30] my client's specifically because it's [14:32] going to be different patterns of what [14:34] good, bad, and and neutral are. Um, and [14:37] so, you know, we we we we start with [14:39] that visualization. A lot of our team [14:41] has just had to get more familiar with [14:42] how do you how do you even start to [14:44] diagnose what the traffic is and then [14:46] how do you contextualize that into who [14:48] our client is. Um, from there then we [14:51] kind of have the categories that that uh [14:53] Brian and Crystal talked about where um [14:55] uh you know we're the the the good [14:57] traffic. We're promoting the the neutral [14:59] traffic. We might be rate limiting or or [15:02] capture challenging so that we add some [15:04] friction to help make that traffic [15:06] identify itself. If if it's not worth it [15:08] to them, then they'll stop. And if it [15:10] is, then we've got more clarity that [15:11] it's good traffic to promote into good [15:13] pathways. And then the bad traffic we're [15:15] blocking, right? But it is it's much [15:17] more of an iterative cycle and and I [15:19] would say one of the things we've had to [15:21] um put an emphasis on is is actually [15:23] even a dialogue with the customers [15:24] themselves, right? Because it's kind of [15:26] a new way of thinking for them. Uh you [15:28] know, usually it comes up because [15:30] they're like, "Hey, our site is down. [15:31] Why is our site down?" And we're like, [15:33] "Man, you just got this massive spike in [15:35] traffic." And they're like, "Well, let's [15:36] fix that." And and you know in the [15:38] interim while especially while you've [15:40] got the grade traffic it can be [15:41] difficult because you're like uh do do [15:43] we increase server resources temporarily [15:46] while we're helping these pathways get [15:48] figured out that has a cost right and so [15:50] so everybody's got this you know um uh [15:53] self-interest in understanding it but it [15:55] but it's complicated and it's it's a [15:57] world that a a lot of people didn't have [15:59] to worry about before. So, so even at [16:01] the front end, we like to uh dialogue [16:03] with our clients and kind of reset their [16:05] expectations like, hey, the the internet [16:07] world of 2026 isn't the internet world [16:10] of 2016 and and what that means even [16:13] from a process standpoint of of you need [16:15] a team that's not just uh going in and [16:18] doing site stuff. You need a team that's [16:20] aware of who's visiting the site, how [16:22] are they using it, what are we going to [16:23] that that should inform UX, but it now [16:25] it should inform the actual security [16:28] rules and and the policies that you're [16:29] putting around your server as well. [16:31] >> It sounds like a much more integrative [16:33] approach than we've had in the past. [16:35] Thank you. One thing I I would love to [16:38] leave all of the viewers of this session [16:40] with. Uh if you all could kind of go [16:43] back to to in your brain when you first [16:45] learned about bots and this problem [16:46] first came up on your radar, what is one [16:49] piece of practical wisdom that you could [16:51] give to yourself or someone else who's [16:53] just starting out with their bot [16:55] strategy? [16:56] >> Take a breath. Take a breath. It's [17:00] moving so fast. And you know to Peter [17:04] John's comment about different site [17:06] personas having different needs whether [17:08] you're ecom whether you're media and [17:10] even inside of that there's kind of sub [17:12] personas and subspaces and what this one [17:15] needs might be different than this one [17:17] and oh my gosh there's so many bots and [17:20] she said over 50% of the internet take a [17:24] breath because the entire internet [17:27] community is dealing with this at the [17:30] same time and we're all dealing with it [17:33] together. So, we all need to learn from [17:37] each other and learn, okay, that worked [17:40] over here. Does it also work over here? [17:43] Maybe, maybe not. And and trial and [17:46] error and see what's going to work for [17:47] your site, for your site persona, for [17:49] your customer. Um, and just allow that [17:54] time to to learn. [17:56] >> Yeah. Maybe just to add to that, I'd say [17:58] um for us it's it's get your [18:00] visualization tool and get familiar with [18:03] that. A lot of agencies are focused on [18:05] on creative and the code and and those [18:07] are important. So it can feel a little [18:09] overwhelming and scary to even be like, [18:11] man, what is this whole new pseudo [18:13] devopsy security world? Who's going to [18:16] do that? Um and I I would say the [18:18] visualization helps reduce a lot of the [18:20] stress because now now you're like it's [18:22] no longer an unknown. It's it's a now [18:25] what do we do with this information? And [18:26] so uh for us, you know, it's it's been [18:29] uh double down on on Cloudflare and and [18:31] their visualization tools. CloudFront, [18:32] whatever your tool is for visualization [18:35] helps reduce the stress both for you to [18:37] then be able to explain to your [18:38] customer, which kind of extends that [18:40] time. So now you've got more time to be [18:42] able to figure out what are we fixing, [18:44] how are we going to fix it, what's next, [18:45] and that's really been helpful to us. [18:48] I think both of you gave the best answer [18:53] already. So I can just echo that like [18:55] you're if first steps, you know, take a [19:00] breath, get visibility, try to [19:03] do the best that's practical, get an [19:05] understanding of what's going on. If [19:07] your organization's large, you know, [19:09] start talking to other stakeholder teams [19:12] that might have a shared interest as [19:14] well. policy if you can uh tackle this [19:16] together as you build a you know your [19:18] own internal POV on what things are [19:21] appropriate and what things aren't and [19:23] then you know the next step from there [19:26] is also I think what Crystal said like [19:28] just um start making some changes and [19:31] start in log mode and see see what [19:33] happens re re-evaluate [19:35] >> great so to double click on that aside [19:38] from practical wisdom is there any [19:40] tooling that you would suggest using or [19:42] that you are using today in order to [19:44] help your own bot strategy in your own [19:47] application. [19:48] >> For us, we, you know, we start with [19:49] something like Cloudflare. Uh that [19:51] really gives us visibility into like the [19:53] geographic of the traffic that the [19:55] access logs and what's actually [19:57] happening so that we can start to build [19:58] out um some of those initial rules, rate [20:00] limits, etc. And and that piece of, you [20:03] know, who your customer is allows you to [20:04] be more specific even with um if if this [20:07] is a product based business or this is a [20:09] a local service business. you know, if [20:12] you're a local service based business [20:13] and you're getting a bunch of traffic [20:14] from uh international traffic, it's a [20:17] lot easier to to make that kind of [20:18] front-end decision on the rate limit [20:20] there. But then what's helpful to us is, [20:22] you know, traffic that's good still has [20:25] opportunity often to be optimized. And [20:27] so understanding beyond even the the [20:29] Cloudflare layer there of okay, I'm [20:31] getting my blog scraped a lot by the [20:34] good stuff that I want, but it's it's [20:36] causing performance issues. And so let's [20:38] loop in New Relic to that to be able to [20:40] understand like what is it about this [20:42] traffic that's causing it to be more [20:44] expensive or why is it not full page [20:46] edge cache uh so that it's having to hit [20:48] the server right and so even uh from WP [20:51] Engine platform we we make heavy use of [20:53] New Relic the the full page caching [20:56] because part of our goal then is to not [20:57] just visualize it but to actually [20:59] implement for the good traffic [21:01] especially. So th those kind of pieces [21:03] work in concert. Um and all those tools [21:06] kind of have a a um a layer that they [21:09] serve uh which is uh kind of that that [21:11] 2011 level of of adjustments that you [21:14] can make beyond even just are we [21:15] blocking it or not blocking it. [21:17] >> So again it's that visibility piece and [21:20] then what are you doing with what you [21:22] just learned from your visibility piece? [21:24] whether the visibility piece is related [21:26] directly to this is the exact traffic [21:28] I'm receiving versus even on the server [21:31] side the traffic has come all the way [21:33] through what is it actually doing to the [21:36] resources on my to use the older term [21:40] hardware right um and and what are you [21:43] going to do about it [21:45] >> spinning off of what we talked about [21:46] earlier you can also [21:49] look at at what others in your industry [21:52] are doing and we kind of talked about um [21:55] you know speaking with other people in [21:56] your company or speaking with peers. Um [21:59] at Cloudflare we're trying to you know [22:01] programmatically aggregate some of this [22:03] data. So you know who's doing what with [22:05] which bot um by by industry vertical. Um [22:10] you know you and if you're working with [22:11] an agency of course like you can ask [22:13] them like they're likely working with [22:15] many other customers um or web [22:18] applications that are similar to your [22:20] own. So they might have some insights on [22:23] what other uh other web application [22:25] owners like yourself are doing and kind [22:27] of help you get started there. [22:29] >> We have a uh a phrase that uh the best [22:32] question should result in another good [22:34] question and and again that kind of [22:36] helps reduce some of the stress of the [22:38] visibility is so that you can ask better [22:40] questions and then usually once you get [22:42] into those uh those should create new [22:45] questions to ask and so that's not a bad [22:47] thing, right? Having the questions isn't [22:49] a bad thing because it means you're [22:50] getting smarter on how do we solve for [22:52] this specific situation for this [22:54] specific client in the new world of bot [22:56] management. [22:57] >> I want to thank each of you all for your [22:59] time today and for bringing this [23:00] expertise into this discussion. Um [23:03] Peter, John, Crystal, Brian, you have [23:05] been awesome to just have a good chat [23:08] around how we're continuing to learn and [23:10] adapt in today's intelligent web. Um, I [23:13] think in general for the community, it's [23:14] great to have these healthy discussions [23:16] about how we're doing things, how we're [23:19] building strategy, and it does not start [23:20] or end here. So, for those of you tuning [23:22] in, I'd really like to encourage you to [23:24] chat with others in your network about [23:26] what they're doing for bots. Um, keep [23:29] sharing your knowledge uh as we all [23:31] navigate this together. This community [23:34] is stronger together than it is when we [23:36] all keep our secrets to ourselves. So, [23:38] let's keep building. Thank you so much [23:40] for joining us.  📄 **Full text:** ![[Full Document Contents/Articles/Good Bots, Bad Bots How to Win With Bot Management.md|Good Bots, Bad Bots: How to Win With Bot Management by YouTube]] ## Highlights - about. Now everybody has to worry about a lot because you want the good traffic, but there's a lot more traffic to deal with. And obviously even from a customer perspective, the moment you've got your site down or you don't know, you know, and and the patterns of what traffic is what and distinguishing that has has just gotten way more complicated over the past even 18 months to where I think everybody's reeling a little bit from, you know, what is the new world that we're in now. >> One thing I think uh several of you mentioned is just making sure that you ([View Highlight](https://read.readwise.io/read/01kr1ty3y0h4syk83y98j44qnk)) - Tags: 👻 ai highlighted - result in traffic being driven to your site. And if you knew about this, you would potentially like decide that you want it. It's just right now you don't know about it. If you offer a mechanism for that operator to like get this same information from you without scraping you, um that incentivizes them to start self-declaring and playing by your your rules. And now you've like pulled some of the problem space out of the gray area and into the like well understood area that makes it easier then to like detect the remaining bots in humans like ([View Highlight](https://read.readwise.io/read/01kr1ty3ytx5pcv4gvjtjrzts4)) - Tags: 👻 ai highlighted - they're like, "Hey, our site is down. Why is our site down?" And we're like, "Man, you just got this massive spike in traffic." And they're like, "Well, let's fix that." And and you know in the interim while especially while you've got the grade traffic it can be difficult because you're like uh do do we increase server resources temporarily while we're helping these pathways get figured out that has a cost right and so so everybody's got this you know um uh self-interest in understanding it but it but it's complicated and it's it's a world that a a lot of people didn't have to worry about before. So, so even at ([View Highlight](https://read.readwise.io/read/01kr1ty42237cajpq71b44c3gv)) - Tags: 👻 ai highlighted - start in log mode and see see what happens re re-evaluate >> great so to double click on that aside from practical wisdom is there any tooling that you would suggest using or that you are using today in order to help your own bot strategy in your own application. >> For us, we, you know, we start with something like Cloudflare. Uh that really gives us visibility into like the geographic of the traffic that the access logs and what's actually happening so that we can start to build out um some of those initial rules, rate ([View Highlight](https://read.readwise.io/read/01kr1ty44zvv0dex1ahx583jeh)) - Tags: 👻 ai highlighted