tl;dr
Most retention metrics look great — loyalty members, WhatsApp subscribers, email subscribers all show higher repeat rates — but this is selection bias, not proof they're working
Incrementality testing solves this by randomly assigning customers to groups so you can compare like for like
Setup takes about 30 minutes in Shopify: a one-time CSV upload for existing customers and a Shopify Flow for new ones
Once in place, use Klar's customer segment comparison to measure the real CLV uplift of any retention initiative
Why retention measurement is broken
Customer acquisition costs have risen by around 50% across ecommerce over the last three years. There's a ceiling to how much you can reduce those costs — and the only sustainable way to counteract rising CAC is through better lifetime value.
The problem is that most ecommerce brands aren't great at retention. On average, fewer than 50% of revenue recurs after the first year, and only about 21% of customers place a second order.
So brands invest in retention tactics: WhatsApp flows, loyalty programmes, print mailings, improved post-purchase experiences. And when you look at the data, every single one of them appears to work. Loyalty programme members have higher repeat rates. Email subscribers have higher LTV. WhatsApp subscribers look even better.
But here's the issue: these groups aren't random. Your best customers — the ones who already love your product — are exactly the ones most likely to opt into your loyalty programme, subscribe to your emails, and sign up for WhatsApp. They were already going to buy again. You haven't caused their higher LTV by enrolling them in a programme. You've just collected them into a group and labelled them.
This is selection bias. And it makes it nearly impossible to know which retention initiatives are actually generating incremental value versus which ones are just attracting your best customers.
What incrementality testing is
Incrementality testing solves this by creating randomly assigned groups of customers. Random is the key word here. When groups are random, any difference in behaviour between an exposed group and a control group is caused by the initiative itself — not by who was already likely to buy again.
The setup is simple in principle: split your customer base into numbered groups (0–9) based on a random characteristic, then expose only certain groups to each initiative. Track the difference. That delta is your real incremental impact.
Setting it up in Shopify
The easiest random characteristic to use is the last digit of the Shopify customer ID. Shopify assigns customer IDs sequentially, so the last digit (0–9) is effectively random across your customer base. You don't need any special tooling — just customer tags.
Existing customers — one-time CSV upload (~10–15 min)
In Shopify, go to Customers → Export and export all customers including tags.
Open the file in a spreadsheet. You'll see a column for tags and a column for customer ID.
In a new column, add a formula that:
Checks whether a tag already exists (to avoid overwriting)
Extracts the last digit of the customer ID
Writes a tag in the format
Incrementality Test - [digit](e.g.Incrementality Test - 3)If an existing tag is present, appends the new tag with a comma separator
Copy the formula down across all rows. Paste as values only (so the formula doesn't carry over).
Save as CSV, go back to Shopify, and import — select Override customers with same email.
That's it for your existing customer base.
New customers — Shopify Flow (set once, runs forever)
In Shopify, create a new Flow triggered by Customer created.
Add a series of If/Else conditions:
If customer ID ends with
0→ Add tagIncrementality Group - 0If customer ID ends with
1→ Add tagIncrementality Group - 1Continue through
2→9
Turn on the flow.
From this point forward, every new customer is automatically tagged and assigned to a group. Total setup time including both steps: around 30 minutes.
Measuring the incremental impact
Once your tags are live, you can run experiments. The logic is:
Introduce a new retention initiative (e.g. a loyalty programme)
Expose only certain groups — say groups 1 and 2 — to that programme
Keep all other groups as the control
After a defined period (e.g. 90 days), compare CLV between exposed and control groups in Klar
Example: Say your loyalty programme costs $7,000/month (software + team). After 90 days:
Exposed groups (1 & 2): average CLV = $92
Control groups (everyone else): average CLV = $73
Incremental uplift = $19 per customer
If you're acquiring 10,000 new customers per month, that's $190,000/month in additional margin from the programme — against a $7,000 cost. That's a real number you can act on.
Real results won't always be this dramatic. But the methodology is sound regardless of the magnitude — because the groups are random, the comparison is fair.
Why this matters now
Every retention tool on the market will show you data suggesting it works — because selection bias makes everything look like it's working. Without incrementality testing, you can't tell which of your initiatives are genuinely driving LTV and which ones are just capturing the customers who would have come back anyway.
Setting this up takes half an hour. It costs nothing. And it's the only reliable way to know what's actually working in your retention stack.