A/B testing email campaigns: what to test and how
Everyone has a theory about what makes an email work. A/B testing settles those debates with data instead of louder voices. Done well, it's a compounding advantage: each test teaches you something about your specific audience that the next one builds on. Done badly, it produces confident nonsense. Here's how to keep it on the right side of that line.
How an A/B test actually works
You create two variants that differ in exactly one element, send each to a random, comparable slice of your list, and after enough responses accumulate you compare a chosen metric and declare a winner. Many platforms then send the winning version to the remaining contacts automatically.
The whole method rests on one discipline: change one variable at a time. Test the subject line, the image and the call-to-action together and a lift tells you nothing about which change caused it.
What to test, roughly in order of impact
- Subject line and preview text — the biggest lever on open rate
- Call-to-action wording, color and placement — drives clicks
- Send time and day — matches your audience's habits
- From name — affects trust and opens more than people expect
- Content layout, length and imagery — influences clicks and conversions
Test the subject line first
The subject line has the largest effect on open rate and is the easiest variable to isolate cleanly, which makes it the natural place to start. Once opens are optimized, move down the funnel to CTAs and content that drive clicks and conversions.
Run a test that's actually valid
Most 'A/B tests' are undone by the same handful of mistakes: samples too small to be meaningful, several changes tested at once, or the test called too early. A result from 40 recipients is a coin flip dressed up as insight.
- Change one variable so the result is attributable
- Use samples large enough to reach a reliable result
- Pick the metric that matches the goal — conversions over opens for revenue
- Give the test time to gather responses before deciding
Make testing compound
A single test is a data point; a testing habit is a strategy. Document what won and why, feed it into the next test, and build a picture of what your specific audience responds to. High-frequency automated flows — welcome, cart recovery — are ideal testing grounds because samples accumulate over time even on a small list.
Frequently asked questions
How big does my list need to be to A/B test?
Large enough that each variant gets a meaningful number of responses — otherwise the 'winner' is just noise. Small lists can still test high-frequency automated flows, where samples build up over time.
What should I test first?
The subject line. It has the biggest effect on open rate and is easy to isolate. Once opens are optimized, move to CTAs and content that influence clicks and conversions.
Why did my A/B test give a misleading result?
Usually too small a sample, testing several changes at once, or ending too early. Change one variable, use adequate samples, and let the test run its course before deciding.
Start sending in minutes
Create a free account, connect your domain and reach your audience across every channel — no credit card needed.
Free plan forever · No credit card required · Set up in minutes