Are You a/B Testing for Maximum Conversions?  ## Metadata - Author: [[WP Product Talk]] - Full Title: Are You a/B Testing for Maximum Conversions? - Category: #podcasts - URL: https://share.snipd.com/episode/c0c061da-8450-42cd-b4d4-3eb3a0a3934f ## Document Notes ## Highlights - **Measure and Gather Evidence** - Measure website changes' impact on conversion rates and sales. - Use A/B testing to gather evidence of the effectiveness of your improvements. Transcript: Katie Keith If you're watching this, then you probably make regular improvements to your website aimed at increasing the sales of your WordPress themes and plugins. But do you actually measure the results of each change and get evidence of the impact on your conversion rate and your sales? So that is where A-B testing comes in. Matt Cromwell This is WP Product Talk, a place where every week we bring you insights, product marketing, business management and growth, customer experience, product development, and more. It's your go-to podcast for WordPress product owners by WordPress product owners. And now, enjoy the show. Katie Keith Hey, welcome to WP Product Talk. I'm Katie Keith, founder and CEO at Barn2Plugins. Matt Cromwell And I'm Matt Cromwell from StellarWP. Katie Keith And today we're talking about an important topic that most WordPress product owners don't do, improving your conversions via A-B testing. Matt Cromwell And that's why we've invited a special guest today named Tom Carlos. Tom, welcome to the show. Tom Carless Hey, thanks for having me on. Matt Cromwell Absolutely. Tom, tell the world a little bit about who you are, what you do, and I don't know, something like your favorite hobby. I don't know. Tell us a little bit about yourself. Tom Carless Ooh, okay. Like you said, my name's Tom Carlos. I created AB Split Test about three or four years ago now. And it's a tool that's designed to help people get evidence around their WordPress updates. A lot of people want evidence to show their bosses, to show their stakeholders, or just to prove to themselves that they're not terrible at this. So that's what we find people love using uh ab split test for um a little bit about myself is i uh i'm a big snowboarder i live in a ski town um and you might be able to see that it is uh currently Snowing quite heavily in the mountain tomorrow so i'm extremely excited for that um that extremely excited to be on here today. So thank you. Cool. Matt Cromwell Where are you in the world? Tom Carless I'm in Whistler in British Columbia in Canada, along with all the other Australians. Matt Cromwell Nice. That's awesome. Nice. Katie Keith So let's talk about why A-B testing matters first. Tom, could you tell us a bit about why people should even be bothering with this? Why can't we just make changes which common sense would tell us will improve our sales and leave it at that? Tom Carless Just love making the common sense updates and then they'll look at their analytics and maybe the analytics have gone a little bit higher and then you kind of pat yourself on the back but Really what i found especially when i was in my previous life as as an agency being able to give really granular results on the updates that you're making it gives you so much more information To sort of continue on making better updates and better updates. So while, you know, updating your homepage with a new catchy heading might sound good, maybe someone else has talked about you in another blog post or somewhere else in the world, and That increases your conversions, but then you associate that increase with you changing your headline or whatever you've done so uh yeah a lot of the time it's mainly just about being Granular enough that your updates actually um can be point pinpointed back to something um why people split test in general is because it makes you more money with the same traffic so You save a lot of money on ads or anything else like that that you're using to get people to your website. ([Time 0:00:04](https://share.snipd.com/snip/22b0d6cf-a86e-4edd-86b4-ef1db0bfef3d)) - **Katie's A/B Testing Journey** - Katie Keith used Google Optimize for A/B testing but switched to Tom Carless's plugin. - She found SaaS A/B testing tools too expensive and now tests videos on sales pages. Transcript: Katie Keith Yeah, definitely. So let's all talk a bit about our personal experiences of A-B split testing. I'll go first because I used to do a lot of this and then stopped. And the reason for that is because Google Optimize, which used to be a free service offered by Google, stopped existing, maybe was it a year or 18 months ago or something. So I used to do it all the time to test everything from should we have a bundle next to our pricing tables, how prominently should we feature the guarantee, should our call to action say Buy now or buy plugin, all sorts of things. And I used to get loads of really great data about what would actively affect the conversions. And it would even give percentages and things like that. And I used to love Google Optimize. And I could do the test myself, even though I'm not a developer. And then it just stopped. So it's taken me until now to get back up and running with it. And I did loads of research of different SaaS products, because I had the idea that because Google Optimize was a SaaS, and not a plugin I installed on my website, I would have rather, In theory, have it outside of my site. But I couldn't find anything that was user friendly, so you didn't have to do code to set up your tests and also affordable they were all everything decent worth hundreds of dollars a Month so I've ended up with your plugin Tom which I think was recommended to me by Ellipsis so we've set that up maybe a month ago and we're currently trialing whether or not it's good to Have a video on our plugin sales pages so we've got our main banner at the top which is text and an image. And we've currently trialing two different videos versus no video. And the video ones are getting more positive results, although the results aren't statistically significant yet, because you need a lot of data to betrust the results. And it's even telling us which of the two types of video are likely to work. So that's really great. So my story will continue of A-B testing, but that's where I am now. ([Time 0:04:12](https://share.snipd.com/snip/2cbabf9a-775d-4fc6-b08c-2b39e7713270)) - **Tom's A/B Testing Experience** - Tom Carless improved conversions by 25% by simplifying the checkout process and adding testimonials. - Videos on the homepage performed worse, possibly due to low quality. Transcript: Katie Keith So Tom, could you give some stories of your own experiences of A-B testing and ideally tests that have had measurable results? Tom Carless Yeah. I mean, the latest test, I do a lot of testing, so it's hard to remember them all. But my latest one we've been doing on A-B split test itself, if you've gone through our checkout, you probably or hopefully didn't notice that you were being tested on. And what we did with that was we removed basically what all the conversion optimization thing says is you remove all the distractions, no header, no footer, everything like that. And we surrounded our checkout form with more trust building stuff like testimonials and just sort of a list of all the features that we've got. And we implemented that on our checkout page. So anyone looking to get the free version of our plugin or the paid version, see that test. And that's given us a 25% uplift in conversions. So for us, a free download is worth sort of 30 plus dollars, depending on how you look at it. So really, any improvement to even the free downloads we can get is a huge uplift for our baseline. So yeah, we implemented that test and it took about two months for us to just get the significant email that you're probably waiting for, Katie. Oh, is that a thing? Katie Keith I haven't received it yet. Tom Carless You'll get an email. It does all the maths for you. So it figures out when you've hit 95% certainty. But yeah, so I mean, that was the latest one that's had a very big positive result. Recently, we tried a similar one to you, Katie, where we put a bunch of videos on the homepage. And we found they performed worse maybe because the videos weren't high enough quality to use but yeah that was that was another good one that saved us from losing 10% conversions per Year so we found that the default version was better than any of our new versions. Going further back in time, like I said, I used to have a WordPress agency and, you know, communications with customers, you can run out of things to talk about after a while, you know, The sort of the monthly check-ins get a little bit bland um so we started getting the question of well what are you doing for me kind of thing and you know we'd explain we were making updates And things like that um and you know it never really lands that well with customers we found so when i started split testing you know we'd say we've updated your home page we've changed Your menu to do this and this and this, and this has increased conversions by 7% or whatever it is. So sort of looking back, those are the ones for me that I found to be the most satisfying are the ones where people say, well, what are you doing and how can we track this back to you? ([Time 0:06:26](https://share.snipd.com/snip/1f3ccc54-1bbb-47dd-9f2f-b5007b529170)) - **Statistical Significance in A/B Testing** - Use Bayesian statistics for A/B testing and aim for 95% certainty. - Increasing certainty beyond 95% takes significantly longer. Transcript: Matt Cromwell That's great stuff. A couple of questions, Tom. Like, especially on that whole issue of statistical likelihood or that we're going to get to a winner. I love how you kind of gamify the plugin, it sounds like, sending a congratulations email when it gets a 95% certainty. I think that sounds super fun. But um is is the the certainty um is that like a a fixed formula that you have or do people have the ability to adjust how certainty is is come to or things like that yeah so you can you can adjust Tom Carless How uh how certain um how certain you want to be um we use it's called bayesian statistics which basically just creates the the classic bell curve out of your your data coming in um the Industry standard is once you're 95 confident then you've got a winner so we're just sort of hard-coded to that if you want you can change it to a higher or a lower number although i wouldn't Recommend going lower um yeah some some people do want a 97 certainty or something like that but every percent after 95 takes a long time so you you end up taking an extra sort of two to three Weeks um per per improvement um so yeah, we use 95%. ([Time 0:13:49](https://share.snipd.com/snip/b84b0bdf-90cb-4777-b73b-a617014feaba)) - **Email A/B Testing** - Test your entire email list for A/B testing, especially with smaller lists (500+). - Prioritize faster data collection over preserving your list, especially if you are a small company. Transcript: Katie Keith We've got a question from Brad Guernsey as well. I have a newsletter, but super new to this type of work. My background is accounting. For AB testing, what percentage of my email list should I test and then time the length? Sorry if you've covered this already. You know, we haven't even talked about email A-B testing, but that is very relevant and something people should be doing as well. Tom Carless Yeah, I mean, email testing is a huge one, especially when you have a list that I would say is sort of 500 people plus, you can start getting sort of real statistically significant results. But I mean, generally, I just say to test on 100% of your list and just split that between the variations that you've created and you think will increase conversions. If you're a huge company like Bantu or something like that that has a lot of email subscribers, maybe it's better to test on a smaller percentage. But in my opinion, I think it's better to get the test going, get the data as fast as possible and move on, rather than sort of an attempt to preserve your list, I guess, is the reason why People don't want to test on their full lists or all their traffic. But generally, I think it's better to get the data quickly and sort of get the information done straight away. ([Time 0:19:58](https://share.snipd.com/snip/b3fcd973-9785-41f8-bf7a-2bfb70e4f1c7)) - **A/B Testing with Low Traffic** - If you have low website traffic, shift your conversion goals further up the sales funnel. - Measure engagement metrics like time on page instead of just final conversions. Transcript: Katie Keith On the A-B testing, we've talked a lot about statistical significance, and that assumes a certain amount of traffic or email subscribers, whichever. What if you're so low volume that you're never going to achieve statistical significance? Is it still worth A-B testing if you're very low traffic, for example? Tom Carless Yeah, well, I would say if you are testing on a really low volume site, the important thing to do is to make sure you're getting conversions. So if you're an agency and you get sort of one contact form per week, that's just, it's going to take you forever. It'll take you years to get to significance, sorry, struggling to talk today. So yeah, when those are the situations, I say it's generally better to bring your conversion further back up the funnel. So maybe it's clicking through to the next page or maybe it's just engagement. So you can say, measure how many people have stuck on the site for 30 seconds versus haven't made it to 30 seconds or whatever the time period may be. You can look at your analytics and see what the average time on your page is and sort of figure out a rough number from there. So you do still need traffic and you still need visits to your page. So if your entire homepage is getting five visits a week, then it's probably not for you. But once you start getting to the hundreds of visits per week, then there's definitely enough data there for you to start testing and optimizing. And a lot of the time, optimizing for engagement for just time on the site is a really good metric for people being interested in your product. You know, like someone that spends 10 minutes on your homepage is likely to be more sort of engaged than someone that clicks through rapidly from one to the next to the next and disappears In 30 seconds. So yeah, it's definitely just matter of bringing your engagements and bringing your conversions further up the funnel. Katie Keith Yeah, that's a good idea. And for example, the A-B test I'm doing is just on one product page at the moment. So that's probably why I haven't had the magical email that you mentioned yet, because it's just one product. But we chose our most highest selling product to get that significant significant i can't say it either that's your fault right to get measurable data quicker um but we if we'd done it Globally on all products we'd probably be there already so i suppose that's a reason to do your whole email list your whole website or something if you've got lower traffic. Yeah, absolutely. Tom Carless We find that you'll need at least a few hundred visits and at least 50, generally 100, 200 conversions before there is enough data to make a decision. If you have 20 variations, then you need more data. But if it's just like a sort of two or three variations, then that's roughly what you need. ([Time 0:25:42](https://share.snipd.com/snip/4089cf34-2e4a-49eb-8598-2296837b16f0)) - **Effective A/B Testing Strategies** - Focus A/B tests on high-traffic pages within your sales funnel. - Use analytics, heatmaps, AI tools, and user feedback to identify areas for improvement. Transcript: Katie Keith It could be a whole page. It could be a tiny bit of wording. So Tom, what sort of advice do you have about how to identify what tests to do on your site? Matt Cromwell Yeah. Tom Carless So, I mean, we've talked about, I think the most important thing to do is to find your funnel and then follow your funnel. So, there's no point doing a split test on your privacy policy page or something like that, that like a tiny percent of your visitors are actually going to. So, it's important to find your busiest pages that funnel traffic into your checkout or whatever it may be. And then after that, it kind of gets into a bit of an analytics thing. So there's three different ways you can go about it. You can look at your analytics or heat maps if you have like a Microsoft Clarity or something like that. And you can see how far down a page people are scrolling and you can see maybe people aren't getting past halfway on my page. And then that gives you an idea of maybe there's something in the middle of our page that needs updating or people aren't actually making it to our pricing page. So maybe we need to do something to make the pricing page more obvious um now that doesn't give you a direct idea of what to do that's just kind of gives you an an idea of the region that you Should be looking at um and this one's kind of hard for me because it's i could talk for hours about sort of finding out what to test um yeah but yeah so after you've found the the rough area That you think has room for improvement that would sort of overall increase your traffic through your funnel um nowadays i just run it through our ai that we've got on the website or built Into the plugin um and that'll give you obviously it's it's ai We've all used chat GPT now. You know that it's 90% of the way there and then it's a little bit weird at the end or in the middle or something like that. But I found that the AI is really good because it just comes in with fresh computer eyes and it gives you some really good unbiased suggestions you may not have thought of at all. Being AI, sort of a third, two thirds of them are terrible, but there's always something that I find the AI will spit out that is something that you've never thought about or you've sort Of written off in your mind for whatever reason before. So you get a lot of good unbiased suggestions that way by using really any of the AIs pointed at your website. Another good way of finding out what to test is like your live chat bubble at the bottom of the site. Is there anything missing from the site? Is there anything you wish you saw on the site? Or just a general question like that that you can have pop up after sort of 30 or 40 seconds. Um and yeah from there you can basically just create a spreadsheet or a we've got a notion doc again on the website you can copy ours if you want um and that just gives you a table of all your Ideas and then you just kind of rank your ideas based on how effective uh you think it'll be so how far up the funnel it is if it's near your home page it's going to have more effect than like I said your privacy policy page and things like that. ([Time 0:34:38](https://share.snipd.com/snip/6db1d1e4-3281-4af1-abb4-277d964262df)) - **Persistence and Simplicity in A/B Testing** - Don't give up on A/B testing; view it as experimentation with no downside. - Keep tests simple, iterate frequently, and learn from both successes and failures. Transcript: Matt Cromwell It. Me personally, I'd say, echo what I said earlier about our approach at Give about diligence and being like having intentional meetings around checking on the results and looking for The next things, like the diligence of checking on your results and thinking of next projects monthly weekly or whatever cadence you feel like is best for you and continuing to iterate Over a period of time. Like it's just, you can't just do like one A-B test and call it done. You really got to be doing it continually in order to see incremental progress towards better results over time. And don't settle, you know, be like, oh, I got a 0.1% increase. I guess that's all I'm going to get. Like, don't think like that. Like, keep trying to get up to two and three and 5% increases. You can do it. So just keep at it over time. Tom, what about you? Best advice for this? This is like elevator pitch best advice. Tom Carless I mean, best advice would be to not give up. It's quite a special, I mean, I would call it a superpower almost, in that you can try an idea, it can be terrible, and then that idea disappears and is gone forever. You try an idea and something great happens and you keep it. So you're like gambling without losing money you know like you can just keep trying keep doing new tests every month or every whatever cadence you want to do um anytime there's a winner You keep that winner forever you get that increased efficiency forever if it's a loser that's great too you know it's a loser you can move on with your life and you don't lose that efficiency Forever. You just lose it for two, three weeks. So, yeah, like you said, I think it's important to just keep testing, keep trying, and also to keep it simple. Like that's why I built Split Test was because it gets really complicated to do more and add bits and code and all of this. If you can just keep it simple follow your funnel find the problems that you're you think your guests are having and then just keep on trying your first split test is probably not going To go that well maybe you get lucky maybe not but if you do three split tests by the time you've done your third one you'll realize how effective it is and you, you know, free money. ([Time 0:47:02](https://share.snipd.com/snip/3119114b-0c0a-4c33-aea0-24e0b23b461f))