Let’s Talk Tooling: Building Your Product Business Infrastructure  ## Metadata - Author: [[WP Product Talk]] - Full Title: Let’s Talk Tooling: Building Your Product Business Infrastructure - Category: #podcasts - URL: https://share.snipd.com/episode/045f17a9-2c23-48b9-af62-23ad667a32f7 ## Document Notes ## Highlights - **Evolving Infrastructure Over Time** - Matt Batchelder shared how fixing small infrastructure annoyances over time significantly improved their process at Stellar WP. - Zach Katz related this to Gravity Kit, highlighting the painful but necessary journey of evolving technical tools. Transcript: Matt Batchelder Maybe a little bit. A lot of homegrown infrastructure and adopting a bunch of third-party tools that kind of make up the maybe duct tapes and twigs that is the thing that runs our business. Matt Cromwell Yeah. I mean, I'll tell you that when Zach brought up this subject as a podcast conversation, I was like, there's only one person I know that really gets excited about this subject. I'm a nerd. Yeah. So I'm really glad you're here. Zach Katz And Matt, why is this important for product owners? Matt Batchelder Well, I mean, as a product business gets bigger, there's a lot of people that have their hands in building the code, pushing it out, pushing products out into the wild, monitoring what Is out in the wild and reacting to it to drive business decisions or answer questions that come in from support. So the infrastructure that we've put in place or products need to put in place really shortens the distance to solving problems or speeds up really boring parts of the build process For a product. So you can actually focus on the really interesting things. Yeah. Infrastructure is so important, but you got to grow it slowly, I think. Yeah. Zach Katz And sometimes I know personally with Gravity Kit, I kind of fell into a lot of the tools that we used and then it took a lot of time to rewind and reconsider and re-implement a better structure. And we're still dealing with technical tooling debt from decisions made in 2014 on how to implement things. Matt Batchelder Yeah, it's such a long tail of problems and annoyances because like the main thing is you want to fix this little thing that's annoying you or taking a lot of time and it's super expensive To make it perfect so you fix these little pieces and your perceptions of things change over time and and the the stuff that gave you a speed up of save an hour three years ago, processes Have changed or it suddenly becomes a thorn in your side and ([Time 0:02:19](https://share.snipd.com/snip/a2c7ec02-b144-4957-942a-d3f00e8c2f64)) - **Docs for Humans vs AI** - AI consumes differently formatted documentation than humans do. - Documentation must be tailored separately for both human readers and AI tools to maximize utility. Transcript: Matt Batchelder You're talking about like process documentation. There's also like the public documentation too, that's in the mix. And now with AI being a thing that can consume this documentation, I think we're on the cusp of trying to figure out some more tooling around massaging the documentation we're already Generating in a way that is AI consumable too. Zach Katz So we're just having a team call earlier today at gravity kit, where we said, if our docs are not step-by convertible into a playwright automated testing script then that might not be A good enough doc but that also is maybe a very dry doc so there's got to be like a different maybe documentation process ([Time 0:05:50](https://share.snipd.com/snip/7f69124b-1fac-476e-b6d5-e7a9016fd401)) - **Slack Bot Automates QA Packaging** - They use a Slack chat bot called StellarBot to trigger packaging of plugin zips for QA. - QA gets instant links to test builds, speeding feedback loops without needing complex builds locally. Transcript: Zach Katz Triggered? Matt Batchelder Yeah, actually, there's a slide that I can jump to. I'm skipping ahead. Let's see. That is in. I have too many slides. Slide 14. Let me jump to that. So these are a couple of examples of. So the one on the left here, this is a look at a chat in Slack, a chat thread, where one of our QA folks have asked for packaging of our event tickets plugin using the feature offline check-in Branch. It's a branch that was in development that she is aiming to do quality assurance for. She sends a very tailored command bot package branch name for plugin abbreviation. Then that will send off to our bot will watch for that. Then it will communicate with ultimately GitHub actions to either grab a zip or package a zip. And then build a link, which our Jenkins server handles signing and orchestrating the delivery of that zip to Slack. And then it will present a link where QA can just click on it and download a dev version of the zip or a final version of the zip if we're getting ready for an actual release. And then they can take that, install that on. We use InstaWP for all of our quality assurance testing and our public demos. ([Time 0:10:47](https://share.snipd.com/snip/406eac72-c13d-4865-a2b8-994d8e1c826b)) - **Scrub Production Data Safely** - Never distribute raw production database dumps to your team. - Always use scrubbing tools to sanitize sensitive customer and purchase data for local testing. Transcript: Matt Batchelder It's all safe. And we can typically recreate problems there with close to production data. And this is something that is important for those who haven't considered it. Zach Katz Please do not just download database dumps and pass them around your support team or whatever. You need to make sure that they are sanitized and scrubbed as Matt was talking about, because there's a lot of information in there that is very compromisable um uh that tool specifically Matt Batchelder Was grown out of the fact that there were cases in our past where we have accidentally emailed customers from local web environments or oh yeah no's, and that's the thing. Zach Katz I try to drop all the tables and like, and do all the good things. I'm still so paranoid when I'm in a local dev environment. The first thing I set up is the like WP block mail plugin. Yes. And it is a multi-site, like a must use plugin. It's my paranoia runs deep based on when I accidentally emailed, you know, thousands of customers for, from a local dev site, not I, but an actually developer working with us on a website. So yeah, it's important to do that stuff. ([Time 0:19:22](https://share.snipd.com/snip/138db887-2e0e-45b0-8dc4-ec29629a9d60)) - **Standardize Local Dev Environments** - Standardize local environments to avoid "works on my machine" issues. - Use container tools like Lando to easily replicate complex development setups for your team. Transcript: Matt Batchelder You a command line tool to do all of that. Then you just define your site with a YAML configuration file which is all plain text and really easy to manage so we started standardizing on that for our internal site so i for for a time And i'm sure many people will experience this you have a new dev join your team or you've got a couple of devs one person is an expert at getting their system running with a site, but local Machines have quirks and people don't necessarily understand how to set up everything exactly. Or, hey, this is working on my machine, but it's not over here. Lando allows us to commit into GitHub a way to spin up a specific site with all the correct versions. So now we can say, Hey, okay, cool. Run this Lando start command and you'll have a site and here's the database you import into it. And all the busy work of figuring out why things are different have drained away because of Lando. ([Time 0:29:34](https://share.snipd.com/snip/2453c673-d1b4-4acb-86c1-44fa8e679907)) - **Centralize Alerts in Slack** - Use dedicated Slack channels to centralize alerts for server, site, and build issues. - This real-time visibility helps coordinate teams and catch problems early. Transcript: Zach Katz The web, like individual websitesal Error Notify is one of the tools we use for that, which is a great tool. I highly recommend it. It connects to Slack and emails. It does webhooks, whatever you want. Blog posts that are updated or created. Talks about we have our GitHub actions whenever a build is created or a pull request is opened or an issue is created. It all goes into Pulse. And so you can actually track things there and have a visual of everything that's kind of happening in the dev process and on the marketing side as well. Matt Batchelder That's pretty slick. We don't have anything that organized for we have a DevOps channel where we do our site deploys. And we have a mix there's some site deploys that can be triggered through chat ops and some other ones that we're, we're trying out a new system that is purely GitHub based, I want to layer Chat ops on top of it. So now you have to look in a couple of places. So it's definitely not as organized as you have. Zach Katz It's nice because sometimes a lot of our team actually does not mute that channel. And then we comment on pull requests like that people didn't know we're coming in or like it's a good way to coordinate. If other coordination fails, it's like a backup uh yeah people know yeah that's nice that's nice for time for a time we had uh we were running periodic reports of pending pull requests Matt Batchelder Uh but as it ([Time 0:37:45](https://share.snipd.com/snip/df28411f-9aa8-404b-9951-1e40756d4cec)) - **Homegrown Tools Scale With Growth** - As your product business grows, homegrown tooling often replaces third-party services. - Tailoring infrastructure to your business needs improves efficiency and control. Transcript: Zach Katz Time. Are there other slides that you really wanted to get to? Matt Batchelder Nothing. I think maybe one slide is worth showing for a moment. And that's slide eight. Can switch to that if we want and this is just the reminder that you know this doesn't happen overnight and the flow at stellar WP well I started at TC so a lot of this is before that gray bar Where so we're looking at a sort of a boxed timeline where the first box is 2015 to 2017 my early days is at the events calendar. We did lots of homegrown tooling and significant use of open source. We weren't as profitable enough to support a lot of the tools that we wanted, so we grew a lot of them. And then as we moved on, the next box is 2018 to 2021. And this is where we kind of shifted to more third-party systems. In addition to adopting additional open source, we de-emphasized homegrown tooling. We tweaked it, but our resources were growing. We had a bigger team. So we were able to bring things focus on development of our products and just adopt some tools. And then as we grew, we were acquired partway through that at Liquid Web and joined what is now Stellar. And that shifted us into 2022 through 2024. And this was actually in, when I started looking back at the overall timeline, it was interesting because we switched away from heavy adoption of third-party services and back to homegrown Stuff because we were at a size and we had the number of developers in our organization to contribute to tooling that was very ([Time 0:53:13](https://share.snipd.com/snip/ff42673d-90e3-4fb8-a495-9f0b6a5ab78c)) - **Grow Infrastructure Slowly** - Grow your product infrastructure gradually to avoid expensive pitfalls. - Focus on pain points and bottlenecks to make meaningful improvements iteratively. Transcript: Matt Batchelder Into uh building their infrastructures out that uh i i think um the the there's a couple of points i think um one take it slow and grow don't try to get out over your skis. It can be very expensive if you're trying to do everything at once because then you have to look at third-party tools and those have costs that might not scale well with your growth. That's the main thing. What I showed on the screen, we didn't get there overnight. It took a decade. Number two, I think this is really important in terms of where you focus, because different teams will have different pain points. And the path that worked for the events calendar and ultimately stellar, the path that worked for Gravity Kit is not going to be the same path that matters to you, depending on the personalities On your team. Focus on the pain points. Any place that you can remove a human bottleneck is a really great place to focus because it enables so many people to do really cool things and work efficiently. And you don't have to build the perfect removal of that bottleneck, but build something that's a little better than today. Not just a bottleneck. Zach Katz I want people to also look for where the balls are being dropped in your processes and where errors are being introduced. And those are also places to look. Yeah. Yeah, that's a really great point. Matt Batchelder I remember packaging when I first joined the events calendar, it would take, to get a very consistently functional package, it would take half a day of massaging change logs and doing All that stuff. Now it takes however long it takes the GitHub action to build, which is like three and a half minutes, maybe. It ([Time 0:59:49](https://share.snipd.com/snip/19d65608-4e61-484d-b7f3-719ae3e284fc)) - **AI-Driven Documentation Workflow** - Use AI with custom instructions to generate documentation efficiently. - First create and approve an outline before generating full docs to maintain quality. Transcript: Matt Cromwell As possible, and then focusing on custom instructions as much as possible that say, give me an outline first and don't write anything until I approve the outline. And then once the outline is approved, then write the article. That has been a game changer for me personally. That's just like the TLDR of what I do in terms of online human readable documentation stuff. Zach Katz And for our side, now when we do PRs, we have our pull requests to GitHub, like trying to introduce changes to our plugin. We have AI automatically generate updated documentation on the PR or new features. We ask it to document new features for different types of users, developers, beginners, advanced users, and have documentation that is catered toward each of them. That's awesome. Yeah. And that's where I was saying that it would be nice to have, like, for our thorough getting started or our thorough, like, here's each one of the settings in this plugin. Here's what it does. Here's how to use it. That's where having the playwright tests that are automatically generated that represents each one of those settings, ([Time 1:02:50](https://share.snipd.com/snip/928cddc3-2968-4902-8bed-98e318cab489)) - **Infrastructure as Team Investment** - Infrastructure is an investment that pays off in team efficiency and product shipping speed. - Ignoring process pain poisons culture and damages company morale and output. Transcript: Matt Cromwell Me, best advice on this subject is to always think of infrastructure as an investment in your team as much as possible. There's, I've had too many conversations where it feels like the conversation goes, oh, we should really build this system to make things easier. But we really got to ship some more product. So we're not going to spend that time. And that that's always attention for sure. Like it feels like spending time on this infrastructure thing is going to pull you away from sales or shipping more product and whatnot. That tension is real. But what you got to recognize is that it's an investment in your team that will pay off dividends in the future and make it easier and better to ship product in the future. So don't get trapped in that, like this, either this or that mentality. Really try to bake in time into infrastructure as much as you can. Zach Katz And like what Matt was saying, if people are in pain about the processes that you use, that is a bad work environment. We work virtually. For your developers, for your support team, for marketing, anybody, if the flow of what you're doing stinks, you are poisoning your culture. So as much as you can do to remove those pains, it helps everybody feel good about their work and how they do it. So yeah, it really is an investment in your team, like Cromo was saying. ([Time 1:04:10](https://share.snipd.com/snip/65c41cf5-c382-4de6-8a28-e1e946bd8a1c))