The Ultimate Guide to Optimizely on Shopify
Want to use Optimizely for A/B testing on Shopify?
In this guide, we cover all the steps that Shopify merchants must take in order to run successful experiments with Optimizely. And if you’re considering other options, we’ll show you why Shogun A/B Testing should be at the top of your list — especially if you’re concerned about cost and ease-of-use.
How to Integrate Optimizely into a Shopify Store
Optimizely’s Web Experimentation product allows you to design page variants and run A/B tests for your site. To integrate this solution with Shopify, you’ll first need to locate and copy the implementation snippet in your Optimizely account:
- Log in to your Optimizely account and select the “Settings” option in the left sidebar.
- Go to the “Implementation” tab of the “Settings” page.
- Click on the clipboard icon to copy the snippet.
Your snippet will look something like this:
<script src="https://cdn.optimizely.com/js/12345678.js"></script>
The next step is to paste this snippet into your Shopify store’s Liquid code:
- Log in to your Shopify account and select the sales channel you want to customize (e.g., “Online Store”) in the left sidebar of the main control panel.
- Open the “…” dropdown menu next to your Shopify theme, then select “Edit code” — this will take you to Shopify’s code editor.
- In the left sidebar of the code editor, you’ll find all your Shopify theme files. Open the “Layout” folder and select “theme.liquid” (you can also use the code editor’s search bar to find this file).
- Paste your Optimizely snippet within the <head> section, just under the <meta> tags (e.g., “<meta charset=’utf-8′>”). Then, save your changes.
This little piece of JavaScript code is what enables Optimizely to work on your storefront. It will load Optimizely Web Experimentation on your site and automatically change the page design for visitors who have been randomly chosen to view a variation (while those who have been randomly chosen to act as the control for the experiment will still see the original version of the page).
And this is why you want to include the snippet as high up as possible in the <head> section of your “theme.liquid” file.
When the snippet is located near the top of the <head> section, Optimizely Web Experimentation will be able to switch over to the variation design while the page is still loading. But if the snippet is located further down the execution path, the original design of the page may be briefly visible before it transitions to the variation— known as page flashing, this glitch will disrupt the user experience, make your business look dysfunctional, and ruin the results of your test.
If you subscribe to Optimizely Data Platform in addition to Optimizely Web Experimentation, then you should also download the Optimizely Data Platform app from the Shopify App Store. This app will allow you to stream your Shopify data into Optimizely, giving you more options for targeting your A/B tests to particular audience segments.
Among other metrics, this app enables you to track Shopify revenue in your Optimizely experiments.
There is one potential issue to look out for, though — revenue totals in Optimizely may be recorded at a lower amount than they are in Shopify. This is because, while Shopify records every sale that goes through, Optimizely may not (e.g., a customer could make a payment but then leave the page before it loads enough for the event to be triggered in Optimizely). Keep this in mind while you’re managing your experiments and evaluating the results.
Setting up Tests for your Store
To create an A/B test with Optimizely:
- Log in to your Optimizely account and go to the “Experiments” dashboard.
- Open the “Create New Experiment” dropdown menu.
- Select the “A/B Test” option.
- You must give your new A/B test a name, and you have the option to give it a description. Then, enter the URL of the page you want to run an experiment on (there’s also a “Target By Saved Pages” option, which is worth setting up if you plan to eventually run multiple experiments on the same page). Once you’re done configuring these settings, select “Create experiment”.
By default, your test will be shown to everyone who visits the page, but you could instead show it only to specific segments of your visitors if you prefer.
Optimizely’s audience builder allows you to use many different types of conditions to create these segments, including:
- Campaign: Target those who are visiting your page with a certain UTM parameter appended to the URL. To set up this type of segmentation, you’ll first need to add the UTM parameters to the links you use for search ads, influencer marketing, etc.
- Referrer URL: Target visitors based on which website directed them to your page (Google, Facebook, Reddit, etc.).
- Browser: You can restrict your test to visitors who are using a certain web browser, such as Chrome, Safari, or Firefox.
- Device: You can also restrict your test to visitors who are using a certain type of device, such as a mobile phone, tablet, or desktop/laptop. For mobile devices, you can even specify whether the brand of the visitor’s device is an iPhone/iPad or not.
- Location: Target visitors based on the country, state, region, or city they are located in.
- Language: You can target visitors based on the preferred language choice in their browser settings as well. This is helpful for when you want to reach all visitors who speak a certain language regardless of where they are currently located.
Multiple conditions can be applied to the same audience segment. In that case, you’ll be able to determine whether all of the conditions or only one condition must be met in order for the visitor to be included.
To create the page variation for your A/B test, open the “Design” dropdown menu in the left sidebar and select “Variations”. You can add variations here, and clicking on one of the listed variations will open up Optimizely’s Visual Editor.
Optimizely’s Visual Editor allows you to:
- Edit elements: Click on any of the existing elements on the page to open its customization settings. Then, you can edit its typography (font size, weight, and color), background (color and image), and border (width, style, and color). CSS code can also be added to customize the element’s styling. And if you want to no longer feature an element on the page at all, you can choose to hide or remove it.
- Add new content: You can insert HTML code to add a new feature to the page. Also, you can upload images from your local file system, and Optimizely Digital Asset Management (DAM) subscribers will be able to upload images directly from their DAM library as well.
- Rearrange elements: Optimizely allows you to determine whether an element is positioned before or after any of the other elements on the page.
Optimizely also allows you to customize both the traffic allocation (this setting determines the percentage of visitors who will see your campaign) and the variation traffic distribution (of the people who see your campaign, this setting determines what percentage will see the original version of the page and what percentage will see the new variation).
These features will help you reduce the risks associated with A/B testing. After all, the changes you make to a page could always end up having a negative effect on user experience — by sending less traffic to the new variation, you’ll be able to limit the reputational damage if your test leads to such unintended consequences.
Now, open the “Track” dropdown menu in the left sidebar and select “Metrics”. This is where you will decide how success is measured for your test.
In order to start your test, you’ll need to add at least one primary metric to track first. You can also add secondary metrics to track as well. There are many different actions and types of behavior that you can set up as a metric, including purchases, newsletter signups, bounce rate, etc.
Once your variation, audience, and metrics have been set up, you’re finally ready to publish your A/B test.
Click on the “Start experiment” button whenever you’re ready (though you should strongly consider clicking on the eye icon next to this button before you publish — this will show you a preview of the visual changes in your variation, so you can double-check that there aren’t any typos or design flaws that would make your test results worthless).
Analyzing Test Results
To access the results of your test, follow these steps:
- Go to “Experiments”.
- Select “Results” or “Manage Experiment”.
- Select “View Results”.
This report will show you how many people are visiting each version of the page, how often your metrics are being hit, and how your new variation is performing compared to the original version of the page.
In addition to these raw numbers, Optimizely also provides some in-depth statistical insights that you can use to make more informed decisions about your test.
Indeed, it’s not always obvious whether the control or the new variation deserves to be the “winner”. If one version of the page has twice as many conversions as the other, then sure, the answer is clear. But most of the time, the results are more muddled.
What if your control has a conversion rate of 10% and your new variation has a conversion rate of 11% — is this improvement due to the changes you made, or could it be random chance?
This is what makes the “Confidence Interval” section of your report so helpful.
The “Confidence Interval” section shows the uncertainty of your test results, displayed as a range of percentages. At first, this range will be wide, and then it will narrow as you collect more data — this is because the more traffic your test receives, the more confident you can be that the results aren’t just caused by variance.
If the “Confidence Interval” range contains 0%, the graphic will stay gray, indicating that you don’t yet have a statistically significant result. If the range shifts far enough to the right that it no longer contains 0%, the graphic will turn green, indicating that the result is statistically significant and positive. And if the range shifts far enough to the left that it no longer contains 0%, the graphic will turn red, indicating that the result is statistically significant and negative.
In other words, the “Confidence Interval” range will let you know whether you have your answer or whether you need to keep your test running for now.
Optimizely also generates graphs for each of the metrics you’re tracking. By default, the “Improvement Over Time” graph will be displayed in the graph section. This visualization makes it clear how much better (or worse) the variation is performing relative to the baseline.
Implementing Changes After a Successful Test
If you’ve determined that your new variation has won the test, then it’s time to publish it for everyone.
In the short term, this can be accomplished by creating a duplicate of your successful Optimizely test that sends 100% of the traffic to the new variation. To duplicate your test:
- Go to the “Experiments” dashboard.
- Open the “…” dropdown menu next to the test you would like to duplicate.
- You’ll see two options here: “Duplicate” and “Archive”. Select “Duplicate”.
Pausing or concluding your initial test and then configuring your duplicated test to only send traffic to the variation will effectively publish your variation for everyone (for as long as the test is running).
For a more permanent solution, you should implement the changes that you tested directly in Shopify. This can be done using built-in tools such as Shopify’s code editor or theme editor.
Once you’ve published your changes to the page through the Shopify admin, you can then conclude and archive the associated A/B tests in Optimizely.
A Lightweight Alternative to Optimizely for Shopify
Shogun A/B Testing offers an extremely lightweight and marketer-friendly way to run experiments on your Shopify store.
With this natively integrated app, you can set up your A/B tests, measure the results, and implement the winning variant without ever leaving the Shopify interface. Instead of needing to learn a whole new page design system, you’ll be able to create the page variants for your test using Shopify’s built-in theme editor. There’s no learning curve at all.
By reducing the amount of effort it takes to experiment, Shogun A/B Testing allows you to get more experimenting done and discover valuable insights in a shorter amount of time.
Shogun A/B Testing is available at a price point that is accessible to merchants of all sizes–making it an option for just about any growing brand.
Shogun also offers advanced features like custom audience segments and traffic splits, so you should be able to set up whatever types of tests you have in mind.
If you’re searching for an affordable, yet highly effective A/B testing solution that’s also easy to implement on Shopify, then look no further.