Proper WordPress Hosting Reviews Decoded: Inside Kevin Ohashi’s Process!  ## Metadata - Author: [[Within WordPress]] - Full Title: Proper WordPress Hosting Reviews Decoded: Inside Kevin Ohashi’s Process! - Category: #podcasts - URL: https://share.snipd.com/episode/765daa35-8ead-40b4-822e-294e247ae58a ## Document Notes ## Highlights - **Data-Driven Hosting Benchmarks** - Kevin Ohashi built Review Signal and its WordPress hosting benchmarks to provide data-driven hosting comparisons. - He’s run benchmarks since 2012 and made them an annual, rigorous process. Transcript: Kevin Ohashi I'm Kevin and I run review signal is, but the main company I've known for running, which is a web hosting review website. Uh, and the thing that really took off within the reviews that I do is called WordPress hosting benchmarks where every, almost every year I would take a large cohort of many of the largest Players in WordPress hosting, and a whole bunch of small ones too, and run them through a rigorous set of tests and publish the results. And that's the main thing I do in the kind of WordPress-y world. I do some other stuff, too. And maybe we can talk about those later, but that's the main one. ([Time 0:00:32](https://share.snipd.com/snip/f5be8c43-e866-4465-9b2a-3763392665fb)) - **Prioritize Uptime And Load Testing** - Focus ranking on real-world impact: uptime and load-handling are the primary determinants. - Record other metrics for context but don't let them dictate overall placement. Transcript: Kevin Ohashi But the main components are, well, there's two kinds of tests in what I do. There's impacting and non-impacting tests. Impacting are the ones that i use to determine the uh kind of ranking award status rankings award status because i don't actually rank them in any order except by gripping with these Ones did well these did pretty well and then there's everybody else um so those are the impacting ones. And then there's the non-impacting metrics where I think these are interesting to record. I think some people might get value out of them or see a way to use them to help either improve their services or figuring out which service might better suit them. But I don't think they are universally applicable to news as a determinant in how we place the companies. So the ones that impact the two, there's two. There's the load testing and the uptime. Can you keep a server online the whole time? And does it respond to large amounts of traffic? If you can pass those criteria, you get a good grade in benchmarks. Then there are the other ones. ([Time 0:06:39](https://share.snipd.com/snip/4a194c4e-ab67-4dc6-a72d-c858c0eb72e5)) - **Beware Single-Server Benchmark Bias** - Single-server CPU/database microbenchmarks can mislead when comparing distributed architectures. - Kevin treats such tests as informative but non-decisive for overall ranking. Transcript: Kevin Ohashi The other ones are stuff like one of them is a plugin. I just threw together. Uh, it's WP, WP performance tester, and it's was designed to test the CPU and test the database of the hosting that you're on and it gives you a neat little score which is nice but there Are problems or limitations i should say with using it which is why it's not an impacting thing uh if you have everything running on one server, your database and the web server, the communication From the web server to the database is a lot quicker. ([Time 0:08:12](https://share.snipd.com/snip/54287198-ddc8-4856-ae9d-a9d50e65f99e)) - **Compare By End-User Results** - Judge hosts by the user's experience, not by specific internal layers like Cloudflare or proprietary CDN. - Use load tests to validate whether the chosen architecture actually delivers under real traffic. Transcript: Kevin Ohashi Cause it's pretty configurable. You know, I, I really care about the end result and that's why the big test is the load testing. It's looking at what happens when you send thousands of users at these sites and what is the experience of those users they're getting. ([Time 0:17:48](https://share.snipd.com/snip/e9231542-18dd-489a-aecc-3c949b0a372e)) - **Why Review Signal Began** - Kevin started Review Signal because he was tired of unreliable, opinion-driven hosting advice. - He aimed to publish transparent, data-backed evaluations instead of subjective recommendations. Transcript: Kevin Ohashi Is it just changing? Like, does nobody trust any of these reviews? How can that be better uh so that's why i started review signal and uh other than a couple attempts that i've seen people make there really hasn't been any serious ones the best attempt I saw was basically a guy copied what i did the original one was scraping scraping twitter and analyzing the sentiment and opinions about companies and yeah he just did he put zero affiliate Links and and he published it i was a little bit hurt because I do have affiliate links cause I need to eat, um, and wanted to try and do something that was honest, transparent and, you know, Afforded a roof over my head. Uh, but he did it, no affiliate links and got, it was wild. Everybody's like, oh, this is great. Like, and whatever. And I'm just like, I just compared the data. I was like, hey, look, it's the same. If you didn't trust me before. But actually, we're friends now. He sold it, I think, to one of those SEO spammy companies. And I would never look at the content there ([Time 0:25:35](https://share.snipd.com/snip/d1ae81a5-9f49-4daf-9bf5-36a2517937c6)) - **Remove Reviewer Bias** - Single opinions are statistically insignificant for general hosting decisions. - Kevin strives to remove his own opinion and treat all companies equally in testing. Transcript: Kevin Ohashi Mean a lot of them are very They're just opinions, a single opinion, a single data point. And it's not that it's invalid. It's that it's statistically insignificant. And so when I thought about what I was doing, my biggest goal in my work is extracting myself out as much as possible. My opinion about any company is the goal is to get rid of it. And I will work with almost any company and put them all through equal treatment. Um, and I've even had complaints about that from some people. It's sort of like, Oh, you know, why do you work with large negatively associated with company? You know, like you do reviews, shouldn't you know better? ([Time 0:29:12](https://share.snipd.com/snip/fe9fcec5-4037-4dda-bf68-4fd9b0a3d4e1)) - **The Portugal.com Acquisition Story** - Kevin acquired portugal.com unexpectedly at auction and had to quickly recreate a 20-year site to preserve SEO value. - He assembled a local team of Portuguese writers and now runs it as an English guide for visitors and residents. Transcript: Kevin Ohashi And the reasoning is real people read this website and are going to go do things. So if I write, I actually, for record rights have written zero articles on there because i'm not portuguese and i don't live there uh and have no business advising anyone about what they Remkus de Vries Should see do you know i am not an expert i i just have to segue into the question that i need to ask now i have to segue why do you own portugal.com um so the first business i ever got involved Kevin Ohashi With was domain names way way back that was the first online business i was in i'm still in that world and at the end of 2021 for some reason i decided to click on one of the dozens of emails That i get you know this is a great domain auction you have to go you know this is the inventory and i clicked it and i saw portugal.com on the list and thought come on that actually doesn't Seem crazy because most of the domains that people are always trying to sell they're either crap or the reserve price is just insanely high that it doesn't make sense that anyone want To buy it but portugal.com was listed there at what i thought was a reasonable reserve and i went huh let me just take a minute and one of the other businesses i'm involved in is called name Bio which is the largest database of domain name sales so if you're trying to do comps so as i i guess i have equity in this company and built tech with it. I was like, I have access to all this. Let me look up what all the countries that I've ever sold for. And you know, most, there's not many of them and most of them sold 10, 20 years ago and for a lot of money higher than this. So I thought, oh, that like a good investment but I don't have that kind of money so I told a friend about it who was in Bitcoin so clearly gambling on a domain name feels like a drop of water In a large bucket and they were like oh yeah let's do it and so he went for it and won the auction and sort of sat there and thought, okay, what now? Well, can I develop it? Cause it'd be a waste. I don't think we expect to win. So it was just like, oh, what now? Okay. Yeah. I don't want to waste it. It's like, yeah, that's kind of like literally the reaction of like, oh, it's like the dog who chases, you know, after some, the car, like, what do you do when you catch it? Uh, yeah. So we got there and it was like, okay, i i'd be up for developing it it's like i'm busy i don't i have other things to do you go for it basically here's the keys uh yeah that's funny and so over About two weeks had to like overnight build the site because it used to be developed too. And so it didn't want to lose the SEO value. So I had to like recreate a 20-year site just unprompted and as quick as possible. And yeah, been running it since. Got a team together, a team of writers who do a great job. They're all either Portuguese or live there, expats living there. And it's just, it's an English language site for people who want to visit, move or live there. ([Time 0:38:29](https://share.snipd.com/snip/aff00090-7c92-4f8d-b8ea-182769d02439)) - **Use Diverse Stress Tests** - Run multiple test types: simulated browsing, static page flood, and WooCommerce flows to cover real use cases. - Include specific WooCommerce tests because e-commerce patterns differ significantly from standard WordPress. Transcript: Kevin Ohashi Would you like to know? Like the history of it or what it is? I can elaborate all you want. Where would you like to know? Remkus de Vries Okay, so I'll give you more direction. So the stress test that you do does a particular set of things. What are they? And yeah, what's it called? What is K6? What is the mix of things that you have? Kevin Ohashi So there's three stress tests and they're each documented. Um, I call one of them, the load storm test, uh, that's named after the previous load testing platform that I used because it allowed more complicated load testing and they were a partner Slash sponsor for many, many years until they went out of business. So that's affectionately, to me at least, known as the Loadstorm test. And what that test did is it simulates somewhat normal WordPress user behavior. So it would go to a homepage. Uh, it would log in as a user and browse around different pages. It's a very basic, this is what most people do on the internet. They go around and click some stuff and sometimes log in. Uh, that's the load storm test. There's a static one, which just hits the home page to generate as much load as possible that is the uh unofficially i call it the did you install caching test uh if you installed any sort Of static caching that says okay somebody's requested that page recently we don't need to go hit anything else and could just serve it uh yeah let's make sure if you fail that test and I don't think any companies have failed it so recently but somebody's gonna go have it tell me oh, of course. Yeah, they have before. There is the occasional failure on it. There shouldn't be at this point. It is 2025. You should have a static cache. But you have to check. Like, you can't just assume. You have to test everything. So we test. I say we. I test that. And the third one is WooCommerce. Yeah. Machines, the many machines. And then there's the WooCommerce tests, which, because we do WordPress and WooCommerce specific. And the WooCommerce is similar to the Lodestorm test, except it runs WooCommerce specific patterns, like add to cart viewing products. Cause it's a different, entirely different WordPress site. Cause it's WooCommerce. Yeah. There's a lot more going on. Yeah. ([Time 0:44:48](https://share.snipd.com/snip/00f03be9-36b6-4eb4-900e-621cd94bd74f)) - **Choose Open Source For Longevity** - Kevin chose k6 because it is open source and programmable, protecting benchmarks from vendor shutdowns or price shocks. - Open-source tooling lets him own the tests and adapt them as needed. Transcript: Kevin Ohashi It is open source. K6 is entirely open source. You can download, use it for free, run it. It's super easy. And it's opinionated. It's for programmers. You write code to emulate the behavior you want. And I like that as a coder. Remkus de Vries Can it do more than stress test? Can it actually do a user cart flow? Like from a user going into a product, add it to cart, check whatever's next, and then, and then, and then whatever step you define is, does it go that far? Kevin Ohashi I believe so. I mean, that's what I do with the WooCommerce test is go to a product page, add it to cart, go to checkout. You can have it. Anything a browser can do, it's designed that it should be able to. It even has extensions that I think you can actually make it use a browser and actually run it through a browser with Selenium, I'm guessing, or WebDriver. I haven't actually done that before through the actual browser. I think it'll be a lot more resource intensive to scale up, but it is capable of it. Remkus de Vries I think I'm trying to remember there's a tool, I forgot the name that allows you to spin allows you to spin up all these types of different browsers that then emulate traffic under the Different browser behavior, you know mostly to simulate real traffic and see where edge cases are. I forgot what the name was. But it sounds very similar. Like you would, by what you just said, K6 could be turned into something like that. Kevin Ohashi I mean, that's the beauty of and it's open source you can turn it into whatever you like uh you can go get the source code and make it you can build extensions uh and that actually kind of Segues into the project that i'm working on now is there's K6, the open source, and then there's K6 Cloud, which was the common monetization. Let's offer a commercial offering to do, provide services around our open source product. So K6 Cloud would let you run the test from the cloud, distribute them, save all the data, visualize the results. So they got, that's who I used to use for the benchmarks every year. They were a sponsor. I really liked the team there. They were great. But they were acquired by Grafana not too long ago and have been integrated into Grafana cloud. And Grafana doesn't have that same, they're a billion dollar unicorn company, not a scrappy startup. And so that- Not at all. Like flexibility and like sponsorship, everything kind of went out the door and the prices increased dramatically. And so one reason I picked K6 is that core of it is open source and you can take it and build whatever you want on top of it and that was always my concern after going through probably two or Three different load testing companies over all this time they keep going out of business or something happens. I picked K6 because it was open source. And, you know, worst case scenario, you still have the source code of that and you can build what you need. ([Time 0:49:02](https://share.snipd.com/snip/a2526fa5-5cfa-4be4-a0cf-bc6d38b88354)) - **Building Orderly Ape After Vendor Changes** - Grafana Cloud acquisition and pricing made commercial k6 untenable, prompting Kevin to build his own platform. - He created Orderly Ape as an open-source k6 cloud replacement to retain ownership and affordability. Transcript: Kevin Ohashi And that's exactly what we've done. So I have a project now called orderly ape, orderly ape.com. And yeah, it's on GitHub.ub it is quietly alpha launched and maybe by the time you publish this interview it'll be more loudly or maybe this is the making it more loudly launched well Let's make it so yeah and so yeah orderly ape is basically a k6 cloud replacement it lets you run your own k6 tests um distribute them across as many hosts as you want of course you have to Set up all of these stuff and it's on your own infrastructure so you own the data it has it uses grafana too because that's also still open source the core and you can see the results so yeah We went and built a i mean it's not as polished as their product it's an open source product and it has been very minimally funded i discovered how difficult it is to raise money for open Source uh still looking for any more companies to step up and help cover the cost if this seems like would you rather have complete ownership over your load testing thing pay like five Or ten grand a month to run it on someone else's cloud so i ([Time 0:53:54](https://share.snipd.com/snip/86f5a0e0-64b7-4c46-92b0-f6392ad13d8d)) - **Sustain Open Source With Sponsors** - Open-source projects need funding even if they aren't commercial businesses. - Seek modest sponsor support from hosting companies to sustain shared infrastructure like load-testing tools. Transcript: Remkus de Vries Is there any thinking in making that into a affordable? Zero. Kevin Ohashi Let's call it none. Like, no, none. Nope. I have watched multiple load testing companies that i worked with that i thought were stable come and go they all seem to follow this trajectory where it's it seems like it's incredibly Difficult to make a business out of load testing and i've seen even more tiny little startups come try and disappear into the ether and i don't understand why other than people don't Seem to want to pay for it at least long term or yeah nobody seems to be able to crack this i also i don't want to work on that like i would rather have it be open source i work with hosting companies Let's open source is great because all that extra value that's being captured is being captured by the people doing stuff i like that yeah i would love for them to have that you know and If they could put a little bit in so that we can make this a little better or contribute or you know whatever that would be amazing but yeah I have no interest in running a load testing business I would rather a dozen people take what we're building and build their own I hope they have grand ambitions and can crack it use us as a base contribute back and make it even better. ([Time 0:56:32](https://share.snipd.com/snip/1fe99fc1-ad2d-4bc8-ad12-69e41a73a0fc))