A client once asked me to test the timing in an email flow. On paper, there seemed to be plenty of traffic. But when I mapped out how people moved through the flow, I could see the test might run for months and still tell us nothing. That’s the trap with email A/B testing: a big top-line audience doesn’t guarantee a useful answer.
Why email A/B tests often fail before they start
The client wanted to know whether the second email in an abandoned cart flow should arrive one day or three days after the first. A few thousand people entered the flow each month, which sounded like enough to test.
But the flow split early. First-time shoppers and returning customers took different paths. Each path split again based on cart value, because higher-value carts received a different message. By the time people reached the email we wanted to test, the audience was spread across four branches.
Those groups weren’t interchangeable. A returning customer with a $200 cart wasn’t the same as a first-time shopper with a $30 cart. Each branch needed its own test, and some only had a couple hundred people a month. Dividing that traffic between two versions would leave very few people in each group.
The test wasn’t going to produce a reliable answer. So we didn’t run it. That may sound anticlimactic, but recognizing when a test can’t answer your question is useful, too. It saved the client months of waiting for data that was never likely to arrive.
Build an email A/B testing hypothesis around a real decision
A test is only worth running if the answer could change what you do next.
Testing two subject lines once, then declaring one the winner by a small margin, doesn’t necessarily give you a lesson you can use next week. If the versions were simply two ideas you happened to write that day, you may not know what caused the difference. One result on its own can be a coin flip wearing a lab coat.
Start with a specific question that matters to the business. For example:
- Does a percentage discount or a dollar-off offer drive more orders?
- Does free shipping outperform either offer?
- Do customer reviews in an abandoned cart email help more shoppers complete checkout?
- Does a clear deadline, such as “Ends Sunday at midnight,” perform better than an open-ended offer?
Write down your hypothesis as an if-then-because statement: “If we put the product name in the abandoned cart subject line, then more people will return and buy, because it reminds them what they left behind.”
If you can’t explain what you expect to happen and why, pause before setting up the test. Even if one version wins, you’ll have a harder time understanding what the result means.
Change one variable and choose the right success metric
Test one thing at a time. If you change the subject line, hero image, and button color together, you won’t know which change affected the result. Testing one element can feel slower, but it makes the outcome more useful.
Choose your success metric before you launch, and make sure it matches the change:
- Clicks can help evaluate elements inside the email, such as layout or button copy.
- Placed orders can help evaluate bigger decisions, such as an offer or send timing.
- Revenue can help show whether a change affected the value of purchases.
Open rate has traditionally been used to evaluate subject lines, but it’s less trustworthy than it used to be. Apple Mail Privacy Protection and automated opens can make the numbers harder to interpret. When possible, look further down the funnel at clicks, orders, or revenue.
Give your email A/B test enough traffic and time
Before launching, check how many people will actually reach each version. My client’s flow test fell apart because its traffic was divided into four branches before it reached the test. The total number of people entering the flow each month hid the small audience in each branch.
More versions divide your audience further. Klaviyo generally recommends keeping a test to four variations or fewer, and using fewer when your audience is smaller. When the list is limited, test bigger, more meaningful differences. Tiny changes are harder to detect.
Klaviyo also provides guidance on how confidently to interpret results. It generally labels a result “statistically significant” when one version has at least a 90% chance of being the winner, and “promising” at 75% or more. It may take roughly 1,800 people per version before Klaviyo can confidently say there’s no real difference. Some sources recommend 10,000 or more when the audience is large enough.
Then give the test time to collect responses. People don’t all open an email as soon as it arrives. A click test may need at least a couple of days; a purchase test may need longer. Klaviyo suggests roughly three to five days for an abandoned cart test, while a flow test can take weeks or months to gather enough data.
Try not to test during Black Friday or a major promotion unless that’s what you’re testing. Customer behavior can be different during those periods.
Log email A/B test results and start where traffic is highest
Don’t change a live test partway through. Klaviyo won’t let you edit a live campaign test, but it will let you edit a live flow test. If something genuinely needs to change, end the test and start a new one instead of changing the conditions halfway through.
Keep a simple test log with:
- Your hypothesis
- The test dates
- How many people saw each version
- The result
- What you plan to do next
Without a log, someone may suggest the same test six months later because nobody remembers it was already run.
Start with messages that more people see, such as your welcome flow, abandoned cart flow, and largest recurring campaigns. They’re more likely to reach useful traffic numbers. Skip transactional messages such as order confirmations, and one-off sends you won’t repeat.
Inconclusive email A/B tests can still teach you something
One of our clients, a clothing brand, tested three sign-up form offers: a 15% discount, a chance to win $200, and a chance to win $2,000. All three performed about the same.
That wasn’t a wasted test. The result suggested the brand could run a single giveaway instead of offering every new subscriber a discount. Increasing the prize didn’t appear to improve performance, either. Sometimes a tie is the lesson.
Most email A/B tests don’t fail because the idea was bad. They fail because the test wasn’t set up to answer a clear question, or because too few people reached each version. Before you launch, make sure you have a real hypothesis, one variable, a metric you’ve chosen in advance, and enough time and traffic to learn something.
And when a result is inconclusive, record that, too. “This doesn’t matter much to our customers” is a useful finding. It frees you to focus on the things that do.
Want to audit what happens after someone joins your email list? I subscribed to 362 ecommerce brands and found that nearly half never sent a welcome email. Get the free breakdown and step-by-step audit checklist at joyjoya.com/frontdoor.
This is the draft only; I haven’t added it to the Content Calendar. If you want it placed there, point me to the specific entry and confirm the edit.
