Key takeaways from the article:
- A/B testing is one of the most effective methods for testing website changes in real traffic with real users. Tests are essential for evaluating the impact of changes and subsequently optimizing the website so it can grow systematically.
- Google Optimize, the tool commonly used for testing, will be discontinued on 30 September 2023 (due to the transition to Google Analytics 4).
- Google currently supports three tools: Optimizely, A/B Tasty, and VWO. The first two include a wide range of features but are more expensive. The third offers a free version and is well-suited for testing in an agency environment, where testing frequency varies across different websites.
Testing changes is non-negotiable
Rolling out changes without prior testing and subsequent optimization is like playing darts blindfolded. You'll only hit the target by chance — and in most cases you won't, leaving website performance the same or even declining. And if you do hit the mark, your conversion rate will grow, but you won't know why or how to repeat it.
If your e-shop or website is your primary sales tool, you can't afford to skip testing. For conversion rate optimization, testing changes and their impact on different parts of the website, A/B testing (or multivariate testing) is the best fit.
What is A/B testing
In an A/B test, you compare two variants — the original and an updated version (if you were comparing two new variants against each other, it would be harder to identify why one performs better or worse than the other). In multivariate testing, multiple elements are tested simultaneously — for example, typography, images, the size of CTA buttons, and so on.
Real users are randomly shown one variant or the other during the test. Defined parameters are measured, such as link click-through rate, time spent on the page, purchases or other conversions, and other metrics. Statistical significance is also taken into account — i.e., whether both variants were seen by a sufficient number of users.
Once testing is complete, you permanently deploy the winning version — or revise and test it again.
Google Optimize is shutting down in September
The tool most commonly used for A/B testing and results visualization at smaller and medium-sized websites is Google Optimize. It was easy to use, integrated simply with Google Analytics, and was available for free.
With the move to Google Analytics 4, however, Google no longer finds it worthwhile to invest in the tool's further development or its integration with the new platform. That's why Google Optimize will be shut down on 30 September 2023. If you've been using it, make sure to back up your data before that date (it's also a good idea to back up the raw data from Universal Analytics) and find a suitable alternative to continue A/B testing.
For now, it looks like Google won't launch a new tool directly inside GA4. Instead, it will invest in integrating existing tools, or possibly opening up an API so other tools can connect to GA4. We've put together an overview of existing platforms that will replace Google Optimize and which Google currently supports.
How to choose a new A/B testing tool
When selecting a new testing tool, proceed systematically in several steps:
- Define your goals, scope, and needs for testing.
- Choose a tool with the features you need.
- Verify the complexity of implementation.
- Find out if there are any testing limitations (number of users, visits, testing duration, etc.). Consider your website's traffic volume and whether the solution is cost-effective.
- Explore the evaluation methods and statistical significance calculations.
- Ensure there will be an internal "owner" of the tool in your company — someone who takes care of its implementation, configuration, and correct usage.
If you're switching from Google Optimize, also check based on your planned tests whether:
- it supports client-side (testing handled entirely by the testing tool) and server-side testing (part of the test runs on the web server, managed by developers),
- it has its own website editor (WYSIWYG or code),
- it allows integration with an analytics tool (such as Google Analytics 4).
When implementing a new tool, keep in mind that a new interface brings a new perspective on the entire testing process. That's why before A/B testing, we recommend first running an A/A test to rule out errors in variant distribution.
There are many tools on the market. The ones Google officially supports and that are financially accessible to medium-sized e-shops are currently three:
Optimizely
A feature-rich content optimization tool that includes A/B testing. It features a visual editor for modifying parts of the website and supports both client-side and server-side testing. Detailed test result outputs are calculated by the Optimizely Stats Engine.
The wide range of features comes with a higher price tag — so consider whether you'll actually use all of them and whether the tool is worth it.
Optimizely currently offers a discount for users migrating from Google Optimize.
A/B Tasty
This tool focuses on A/B and multivariate testing of websites, apps, tools, and software. It offers client-side and server-side tests, a no-code and low-code website editor, and the option to use your own audiences or pre-defined ones from the tool. It also includes more advanced statistical significance calculation methods.
Given its feature set and higher price, it's again better suited for larger projects. As a Google partner, it also offers a discount for users migrating from Google Optimize.
VWO
Unlike the first two mentioned, it offers a basic free tier where you can test up to 50,000 users per month at no cost. It's therefore well-suited for occasional testing or testing across different projects — for example, in an agency environment where implementing a robust A/B testing platform isn't cost-effective.
Keep in mind, though, that the basic version doesn't support server-side tests, which are set up by developers to test more complex changes, new sections, or website features. Client-side tests are quick and easy, but occasionally produce visual glitches in the editor — for example, a brief flash of the original text or button. Monitor the appearance of tested variants carefully.
Otherwise, VWO offers the same features as Google Optimize. It has a visual editor for modifying certain parts of the website, allows importing segments from some third-party tools, and offers a better test evaluation interface.
The ideal Google Optimize replacement
For these reasons, we consider VWO the best replacement for Google Optimize — especially if you test across different projects at varying frequencies and a more expensive tool isn't cost-effective.
However, if you plan to test and optimize consistently on a larger project, try Optimizely or A/B Tasty instead. Alternatively, there are other top-tier tools on the market, such as Conductrics, which combines testing with user research and can accelerate performance improvements even faster.
Or have a custom tool built to your needs, if you have capable developers on hand. Measurement can be integrated with Google Analytics 4, and you can track performance impact yourself. Bear in mind, however, that this requires significantly more effort both for developing the tool and for evaluating results.
