How to A/B Test Cart Offers on a Low-Traffic Shopify Store

A low-traffic store can A/B test cart offers, provided it tests big differences, changes one thing at a time, and lets the test run for whole weeks until the result is clear. Small stores get into trouble by testing small tweaks and stopping at the first encouraging day. This guide covers what to test first, how to keep the comparison fair, how long to wait, and why the deciding number is profit per visitor and not order value.

What should you test first?

Test the decisions that change what the shopper gets, not how it looks. With limited traffic you can only afford a few tests a year, so spend them where the two versions could plausibly produce very different orders. Three are worth the time, in this order:

1. The threshold. Where the free-shipping or reward threshold sits decides how many shoppers reach for it and how many simply get it. Try two values that are clearly apart, not a few dollars either side of each other. Our guide to calculating a free shipping threshold gives you sensible candidates.
2. Gift versus discount. At the same threshold, offer a free gift in one version and a money-off reward in the other. They cost you different amounts and attract different shoppers. See how to offer a free gift with purchase for the set-up.
3. The offer product. Keep the offer slot the same and change which product sits in it.

Button colours, wording and icon styles come last. On low traffic the difference they make is usually too small to detect.

Why change only one thing at a time?

Because otherwise you cannot say what caused the result. If version B has a higher threshold and a different gift and a new headline, and it wins, you have learned that the package works. You have not learned which part to keep.

One change per test also keeps the next test useful. The winner becomes the new baseline, and the next change is measured against it.

Why must each visitor keep the same version?

A shopper who sees a $60 threshold on Monday and a $75 threshold on Tuesday is in neither group, and they have also just watched your offer get worse. Both the data and the customer's trust suffer.

A proper test assigns each visitor to a version once and keeps them there for the length of the test. Check that your tool does this, and that it splits visitors at random and not by time of day or traffic source.

Do not run a test by switching the offer yourself on alternate days or weeks. That compares days, not offers, and days differ for reasons that have nothing to do with your cart.

How long should a test run?

In whole weeks, for a length you decide before you start. Order patterns differ between weekdays and weekends, so a test that runs from Wednesday to Monday gives one version more weekend than the other would get in a full cycle.

On a low-traffic store, expect to think in weeks or months, not days. Some practical rules:

• Run at least two full weeks, and longer if orders are few.
• Avoid starting just before a sale, a product launch or a large email send. If one happens anyway, both versions see it, but the period stops being typical.
• Run one test on the cart at a time. Two overlapping tests share the same small pool of orders and blur each other.
• Do not change either version while the test is live. If something is broken, fix it and restart the clock.

Why wait for a clear result?

Because small samples swing. With a handful of orders in each group, a single large basket can put one version ahead for days. Stop then, and you have picked a winner by chance.

The discipline is to decide the end date in advance and not act on the running score. Look in on the test to check nothing is broken, but do not call it.

If the test ends without a clear winner, that is a result too. It tells you the two versions perform about the same, so choose the one that is cheaper to run or simpler for the shopper, and move on to a bigger question.

Marka Cart Drawer follows these rules in its A/B tests on offers and designs: each visitor keeps seeing the same version, and a winner is called only when the result is clear. Tests are available on its paid plans.

Which number decides the winner?

Profit per visitor. Order value alone flatters any version that gives more away, and conversion rate alone ignores what each order earned.

Profit per visitor is the profit from all orders in a group, after the cost of goods, shipping and the reward itself, divided by every visitor assigned to that group, including those who did not buy.

Here is an example with made-up round figures. Say each version is shown to 1,000 visitors.

• Version A, lower threshold: 30 orders, average profit of $20 an order. Total $600, or $0.60 per visitor.
• Version B, higher threshold: 27 orders, average profit of $24 an order. Total $648, or about $0.65 per visitor.

Version B has fewer orders and still earns more. Judged on conversion rate, A wins; judged on profit per order, B looks further ahead than it is; profit per visitor weighs both. Note also that a gap of three orders on this volume is the kind of difference chance produces, so this test would need to keep running.

For the full set of numbers to watch, see our guide on measuring whether a cart change raised order value.

A short checklist

1. Choose one question: threshold, gift versus discount, or offer product.
2. Make the two versions clearly different, and change nothing else.
3. Confirm each visitor is assigned at random and keeps one version.
4. Fix the length in whole weeks before you start, away from sales and launches.
5. Leave both versions untouched until the end date.
6. Compare profit per visitor, and treat a narrow gap as no result.
7. Adopt the winner as the new baseline, then pick the next question.

Testing is how you find your own numbers for the advice in our guide to Shopify cart drawers and order value. If you would like the splitting and the waiting handled for you, the testing feature is described on the Marka Cart Drawer page.