Try PagePilot.ai 100% FREE! No credit card required.
Cover image for How to A/B Test Shopify Product Pages: Setup, Metrics, and Reading Results

How to A/B Test Shopify Product Pages: Setup, Metrics, and Reading Results

• 12 min min readMilos M - Author

A/B testing a Shopify product page means showing shoppers two versions of the same page to find out which one performs better.

One version is your existing product page, known as the control. The other is a modified version, called the variation. Shoppers are randomly assigned to one or the other during the same period, and you compare how many completed the action you're testing, i.e. making a purchase.

A testing tool can handle which version each shopper sees, but you'll still need to make sure purchases are tracked correctly and that there's enough evidence to trust the results.

There's also an important difference between a proper A/B test and simply updating your page to see what happens.

Say you rewrite your product description on Monday and sales increase compared with the previous week. Was it the new copy? Maybe. But you might also have received better traffic, offered a discount, or benefited from a temporary increase in demand.

An A/B test gives you a much better way to isolate the effect of your change, because both versions are tested at the same time under comparable conditions.

Key Takeaways

  • Test a customer problem, not a random design preference. Repeated questions about product fit are a better starting point than changing button colors without a reason.
  • Randomly assign eligible shoppers at the same time. Returning visitors should keep their assigned version wherever the tool supports it.
  • Calculate the sample before launch. A minimum visitor count displayed by an app isn't proof that the experiment has enough power.
  • Check the actual outcome being measured. Add-to-cart events are not completed purchases, and Shopify sessions aren't necessarily unique experiment visitors.
  • An inconclusive result is legitimate. It means the evidence doesn't resolve the question, not that both versions are equivalent.

What Makes Shopify Product Page A/B Testing Reliable?

A reliable A/B test compares two versions of a product page under the same conditions, with a clear goal and a way to measure the results. You should know exactly what you're changing and what would count as an improvement before the test starts.

Let's say a Shopify store sells a compact travel backpack. Customers keep asking whether it'll fit under an airline seat. The dimensions are already on the product page, but they're buried near the bottom, where shoppers might miss them.

The merchant suspects that making those measurements easier to find could help customers decide whether to buy.

Here's how they could test that idea:

Version A is the existing product page, with the dimensions near the bottom.

Version B moves the verified measurements beside the purchase options and updates the opening description to make the backpack's size clearer.

Everything else stays the same. Both pages sell the same backpack at the same price, with the same variants and checkout process.

A testing tool then randomly assigns eligible shoppers to one of the two versions. Ideally, someone who returns to the store should keep seeing the version they were originally assigned.

That's important because five visits from one shopper aren't the same as five different shoppers. For this test, we're measuring results by identified visitor, although the tool's ability to recognize returning shoppers will affect how accurately that works.

The merchant also needs to decide what success looks like.

In this example, the main metric is the percentage of assigned visitors who buy the backpack within seven days of first seeing their assigned page.

We're using seven days as an example of a measurement window chosen by the merchant, not suggesting it's a built-in feature of PageFly or EComposer.

Now the merchant has a specific question to answer: Does making the backpack's dimensions easier to find actually lead to more purchases?

And because both versions run at the same time, under comparable conditions, the results are far more useful than simply changing the page and hoping sales improve.

Which Product-Page Change Should You Test First?

Start with something that's making customers hesitate or preventing them from buying. Support questions, product reviews, and onsite searches can help you spot those problems. You can also look at where shoppers tend to leave your product page or abandon the buying process.

Take our backpack example. Customers keep asking about its dimensions, so there's a good reason to test whether making that information more visible leads to more purchases.

Changing the Add to Cart button from black to green? There's no obvious reason to expect that to solve the problem.

You don't necessarily have to change just one thing, either.

For example, you could rewrite the opening description to emphasize the backpack's size and add a measurement panel beside the purchase button. Both changes address the same customer concern.

Just remember that if Version B performs better, you won't know which of those changes made the difference. You'd need another test to find that out.

The same thinking applies to other products. Maybe shoppers don't understand how a product works, or they're struggling to tell two variants apart. Perhaps they keep asking about delivery because the shipping information is difficult to find.

These are all reasonable starting points for an A/B test because you're responding to something customers have actually shown you.

PagePilot's product-page CRO checklist can help you identify other areas worth improving. A/B testing then helps you find out whether those changes make a measurable difference.

One exception: don't A/B test something that's clearly broken. If selecting a medium backpack adds a large one to the cart, fix it immediately. You don't need an experiment to establish that customers should receive the product they selected.

How Much Traffic and Time Does a Shopify A/B Test Need?

There's no magic number of visitors or days that makes an A/B test reliable. How much traffic you need depends on your current conversion rate, the improvement you're hoping to detect, and how confident you want to be in the results.

Let's go back to our backpack example.

Say the product page currently converts 3% of eligible visitors into buyers. The merchant wants to know whether the new page can increase that rate to 3.6%.

That might sound like a small difference, but it's actually a 20% relative increase in conversions, or 0.6 percentage points.

In testing, this target is called the minimum detectable effect (MDE). It's the improvement you're designing the experiment to have a reasonable chance of identifying, not a promise that the test will detect every smaller change.

So, how many visitors would our backpack test need?

Using a conventional fixed-duration A/B test with traffic split evenly between the two versions, 80% statistical power, and a two-sided 5% significance level, we'd need approximately:

  • 13,900 visitors to Version A
  • 13,900 visitors to Version B
  • 27,800 visitors in total

That calculation assumes we're measuring independent visitors correctly. It's an estimate for this particular experiment, not a minimum traffic requirement for every Shopify store. Statsig's power-analysis guide explains the statistical assumptions behind these calculations.

Now let's put those numbers into perspective.

If the backpack page receives 4,000 eligible visitors a week and everyone enters the experiment, collecting enough traffic would take roughly seven weeks.

If only half of those visitors enter the experiment, you're looking at around 14 weeks instead.

And because our example measures purchases within seven days of a visitor's first exposure, we also need to wait for the last visitors' purchase windows to finish before analyzing the final results.

The test should cover normal weekday and weekend shopping behavior, too. You don't want to judge a page's everyday performance entirely on traffic from a major sale.

For smaller Shopify stores, this creates a practical problem. You might simply not have enough traffic to detect a modest improvement within a reasonable timeframe.

That doesn't mean you should run the test for three days and declare a winner after a handful of purchases. It means you may need to test a change expected to have a larger impact or reconsider whether the experiment is worth running at your current traffic level.

One Shopify reporting detail is also worth checking.

Shopify changed how it measures sessions between September 21 and 23, including how sessions spanning midnight are counted and how bot traffic is filtered.

As a result, conversion rates from before and after that change may not be directly comparable.

When estimating your sample size, use a recent baseline measured consistently. And if your testing tool counts individual visitors while Shopify reports sessions, don't combine those numbers into one conversion-rate calculation.

That Shopify reporting change doesn't automatically invalidate a visitor-level A/B test measured independently. You just need to make sure you're comparing the same thing throughout the experiment.

See Shopify's session-measurement update.

Before You Launch: Document Your Test Plan

Before setting up your A/B test, write down what you're changing, why you think it might help, and how you'll know whether it worked.

Be specific about who will see each version and what you're measuring. For our backpack example, that means deciding which visitors are included, how they're assigned to a page, and what counts as a purchase.

Use your store's recent conversion data to estimate how much traffic you'll need. Decide which metric matters most, how you'll monitor revenue or profitability, and when the test should end.

Keep those notes so you can compare your original plan with what actually happened.

When the results come in, you'll have a much easier time deciding whether the new page genuinely performed better or whether something else affected the outcome.

And if tracking broke, traffic fell short, or you made significant changes halfway through, you'll know why the results might not be reliable enough to act on.

How Do You Split Traffic Between Product Page Variations?

Use a testing tool that can randomly allocate eligible visitors and preserve their assignment well enough for the intended analysis. Merely creating two product-page URLs doesn't do that.

Before choosing a provider, confirm three things:

It can show the appropriate page version to each assigned visitor.

Track a completed purchase back to that assignment.

Support your defined measurement window.

Why? Because a visual page builder and a testing statistics dashboard do not automatically provide all three.

Preparing a Product Page Variation With PagePilot

PagePilot's editing workflow lets merchants change product-page copy, button text, and images before importing a page into Shopify. They can also review product variants, prices, and SKUs.

For our backpack example, the merchant could use PagePilot to prepare a version with clearer opening copy and more prominent measurements. The original product information would still need to be checked, particularly the prices and variants that should remain identical across both versions.

That prepares the page for testing. It doesn't randomly assign visitors or measure which version generates more purchases. Those functions still require a compatible experimentation and measurement setup.

PageFly: Participation Isn't the Same as the Version Split

PageFly's A/B testing guide documents a control/variation workflow and two allocation settings.

Page traffic tested determines how many eligible visitors enter the experiment. Page versions determines how those participating visitors are divided between A and B.

If half the eligible traffic participates, and the test splits that half evenly, each version receives approximately a quarter of all eligible traffic. The other half is outside the randomized comparison; don't add those people to the control group when calculating the result.

PageFly documents goals such as add-to-cart rate, product view rate, and PageFly event conversion rate. Its published instructions don't establish that these goals deliver our exact visitor-level, seven-day backpack-purchase outcome.

Check event definitions and the available attribution setup before selecting the goal. If the tool cannot produce the required outcome, use compatible measurement or choose a different testing setup.

PageFly also offers Pause the test when a winner is found, which can stop a test at a configured win-probability threshold. That setting should not be treated as equivalent to the fixed-horizon rule in our example.

Disable automatic pausing for that plan and arrange measurement and final analysis that actually follow the chosen method. The provider describes its own win-probability approach. Don't substitute its threshold for a conventional p-value.

EComposer: Check Returning-Visitor Behavior

EComposer's A/B testing documentation covers page, URL, and template testing, along with setup involving GA4 and a Shopify Pixel. It reports visitor, conversion, and revenue metrics, but the exact purchase attribution required by our example still needs verification.

Its One time only setting retains a visitor's assigned version on subsequent visits. Every visit may show a different version on a return. For a visitor-level purchase test, use stable assignment rather than deliberately changing the shopper's experience between visits.

Neither browser persistence nor cookies guarantee perfect identification across devices. Consent settings can also affect tracking. Test these limitations in your configuration instead of assuming every purchase can be matched to an exposure.

As of November 13, 2026, PageFly Optimize advertises A/B testing from $39/month, while EComposer Pro is $49/month or 39/month equivalent when billed annually. Confirm the plan and available features in the app before subscribing.

What Should You Check Before Launching?

Before starting your A/B test, make sure both product pages work properly and that purchases are being tracked correctly. Otherwise, you could end up comparing a working page against a broken one rather than finding out whether your changes improve sales.

For our backpack example, go through both versions as a customer would.

Try selecting every available size and color. Check that the correct price appears, discounts work, and inventory messages are accurate. If you offer subscriptions, test those too.

Then add the backpack to your cart and complete a test checkout using Shopify's supported testing method. Repeat the process on mobile, where customers may experience the page differently.

Unless you're specifically testing shipping or payment options, those should work the same way on both versions.

Next, make sure your tracking captures the purchases you actually want to measure.

Use controlled test visits and orders to check a few things:

  • Can you tell whether a visitor was assigned to Version A or Version B?
  • Does buying the backpack count as one purchase, without duplicate events?
  • If someone buys a different product, does that stay out of the backpack's purchase count?
  • Can your testing setup capture purchases made within the full seven-day window, even for shoppers who first visited near the end of the experiment?

You'll also need to account for customer privacy settings.

As Shopify explains in its privacy guidance, customers' consent choices can affect which analytics data you're allowed to collect.

Test tracking under the consent settings that apply to your market, and make sure both versions follow the same rules. Automated traffic should also be filtered consistently so bots don't inflate your visitor counts.

One more thing: check how your testing tool handles URLs.

If both versions use separate URLs, search engines may encounter both pages. Google Search Central recommends using a canonical link to identify the preferred URL and temporary 302 redirects when redirects are needed.

That doesn't mean every test variation should automatically receive a noindex tag. Tests that display different content on the same URL don't have the same requirements.

Once both pages work, tracking checks out, and visitors can complete purchases without problems, you're ready to launch the experiment.

Launch QA Checklist

  • Both pages show correct prices, selected variants, stock, discounts, and shipping.
  • Cart, checkout, and any subscription or bundle controls work on desktop and mobile.
  • Exposure and purchases are recorded once and attributed to the right assigned visitor.
  • The seven-day purchase window and product-specific outcome are actually supported.
  • Consent, bot filtering, assignment persistence, and alternate URLs have been checked.
  • Sample target, traffic split, and stopping method are recorded before launch.

Which Metrics Should Decide the Winner?

The most important metric is whether more shoppers actually buy the product you're testing. But before declaring a winner, you also need to check whether those extra purchases are good for your business.

For our backpack experiment, we're measuring the percentage of eligible visitors who purchase the backpack within seven days of first seeing their assigned page.

That gives us a clear comparison between Version A and Version B.

You should still look at other metrics, but don't confuse them with the main goal.

For example, add-to-cart rates can tell you whether more people are showing interest. Checkout initiation can help you see whether shoppers are moving closer to a purchase. Both are useful for understanding customer behavior, but neither proves that the new page generates more sales.

You'll also want to look at revenue per visitor across the entire store, not just backpack sales.

Imagine Version B convinces more shoppers to buy the backpack, but those same visitors end up spending less on other products. The backpack's conversion rate might improve while your overall revenue suffers.

Average order value can help you spot changes in how much customers spend per order. And if you have reliable product costs, contribution per visitor can give you a better idea of whether the variation is actually making you more money.

Finally, remember that refunds and returns don't always happen during the experiment. You may need to check those later before deciding whether the improvement was commercially worthwhile.

The goal is to find out whether your change produces more valuable purchases without hurting the rest of the business.

Hypothetical Results: More Purchases, but No Clear Winner

Assume the fixed-horizon sample is complete, the groups pass assignment and tracking checks, and every seven-day outcome window has closed. These figures are entirely hypothetical.

The variation produced 56 more purchasing visitors. Its conversion rate is 0.4 percentage points higher, or 13.3% higher relative to the control. These statements describe the same observed difference in different units.

But the conventional two-sided comparison gives a p-value of approximately 0.057, just above the predeclared 0.05 threshold. The approximate 95% confidence interval for the difference runs from -0.01 to +0.81 percentage points. That range includes zero and still allows for a commercially meaningful improvement.

The new page might be better, but this experiment hasn't resolved the question under its planned decision rule. It would be incorrect to call B the proven winner, or to conclude that A and B perform equally well.

When Should You Stop the Experiment?

Decide when the test will end before you launch it, and stick to that plan unless something goes wrong.

For our backpack example, we're using a fixed-duration test. That means you shouldn't keep checking the results and end the experiment the moment the p-value drops below 0.05. Doing so increases the risk of declaring a winner when there isn't one. Sequential and Bayesian testing methods allow different approaches to monitoring results, but they come with their own rules.

While the experiment runs, make sure visitors are being assigned to each version as intended.

For example, if you're testing a 50/50 split and one version receives far more traffic than expected, you may have a sample ratio mismatch. That could mean something is wrong with how visitors are assigned or how their activity is tracked.

A 51/49 split isn't necessarily a problem, though. It depends on how many visitors you're testing. Microsoft's research on sample ratio mismatch explains how to tell normal variation from a genuine issue.

Keep track of anything else that could affect the results, including tracking changes, major promotions, or other experiments running at the same time.

And what if Version B's purchase button stops working halfway through the test?

You haven't learned that customers prefer Version A. You've learned that Version B was broken.

That's a reason to investigate the test, not declare a winner. Once you've fixed the problem, you may need to start the experiment again.

What Should You Do With the Results?

Once your A/B test ends, decide whether there's enough evidence to use the new page, keep the original, or run a different test. Not every experiment produces a clear winner, and that's perfectly fine.

Let's go back to our backpack example.

The merchant has collected the planned number of visitors, but the results don't provide strong enough evidence that Version B increases purchases.

That doesn't mean the new page is worse. It simply means the test was inconclusive.

The merchant could keep the original page and look into another problem customers are having. What they shouldn't do is leave the experiment running indefinitely, hoping the numbers eventually favor Version B.

And if something went wrong during the test, such as broken tracking or an unequal traffic split caused by an assignment issue, the results may not be reliable at all. In that case, fix the problem before considering another test.

Whatever the outcome, save a record of what happened.

Keep your original hypothesis, screenshots of both pages, test dates, planned and actual visitor counts, conversion results, and the final decision. Include any revenue or profitability metrics you monitored.

This is especially important if you're using PageFly. According to PageFly's documentation, resetting an A/B test deletes the data you've collected. Export or save your results before doing that.

If the evidence is strong enough to justify using Version B, publish it and go through the live buying process again. Check that the correct product, price, and variants appear and that checkout still works.

A successful experiment doesn't guarantee that nothing will break when you publish the changes.

And remember, not every product-page improvement needs an A/B test. You can find other opportunities to improve your store in our Shopify conversion rate optimization guide.

Build the Variation, Then Test It Properly

Once you've decided what to change, PagePilot's AI product-page generator can help you create an updated Shopify product page without starting from scratch.

Before testing it, check that the product details are accurate, the images match what you're selling, and customers can select the correct variants at the advertised prices.

PagePilot helps you build the new page, but it doesn't run the A/B test for you. You'll still need a compatible testing tool to split traffic between the two versions, track purchases, and collect the data you'll use to compare them.

Those are different jobs, and both matter if the final decision is going to mean anything.

Last updated: