Test your emails

A test tells you which of two emails works better, if it's fair: both versions sent at the same moment to a random split of your list, one thing different, judged by one measure you chose before sending. On most small lists a test can't see small differences, so test the things that could double a result, and repeat them.

The short version

• One change, two versions, the same moment, a random split. This month against last month isn't a test.
• Choose the measure before you send: clicks, or the goal itself, orders, bookings or replies. Never opens.
• Know what your list can see. With 500 people a version, a test only spots a click rate nearly three times the other.
• On a small list, test big things: who gets it, the offer, the message. Not a button colour or one word.
• Repeat a test and add up the results. One win is an anecdote; a version that keeps winning is a pattern.
• Set the size and the finish before you start, and never stop early because one side looks ahead.
• Test the first welcome email first. Every new subscriber gets it.
• Check every email before it goes: the links, the unsubscribe, personal fields, the subject and preview, your terms, and a test send.
• Keep a test log.

What makes a test fair

HubSpot's guide to A/B testing (about 2012) set out the rules every later guide repeats: test one thing at a time, split the list at random, send the versions at the same time, and decide in advance how sure you need to be. On timing: "If you were to run test A during one month and test B a month later, you wouldn't know whether the changed response rate was a result of the different template or the different month."

Before-and-after isn't a test. It mixes the change with the season, the news, the list's growth and the offer. Only versions sent at the same moment, to people chosen by chance, isolate the change. As one reviewer put it of Kohavi, Tang and Xu's Trustworthy Online Controlled Experiments (2020): "The only thing worse than no experiment is a misleading one, because it gives you false confidence!"

The words:

• The control is your usual version; the challenger is the new one.
• The variable is the one thing that differs.
• The deciding measure is the one number, chosen before sending, that decides which won.
• Test, then send the winner: the test goes to a slice of the list, and the winning version to everyone else.
• A false winner is a version that looked better only by chance.

Judge by clicks, not opens

• Litmus, the day Apple's privacy change shipped in September 2021: "A/B testing subject lines (or anything else) using opens to determine the winner or to automatically send out the winner don't work as well anymore."
• Apple's setting now touches roughly 55 to 60% of recorded opens (Litmus, September 2026): Measure what matters.
• Subject lines too. With the rest of the email the same, a difference in clicks can only come from who opened, or from chance. So clicks judge a subject line fairly.
• Where a tool lets you choose how the winner is picked, Mailchimp for one offers open rate, click rate or total revenue, choose clicks or revenue.

What your list can see

The smallest click rate a two-version test can reliably tell apart from a 2% click rate, worked out with the standard sums for comparing two rates (5% significance, 80% power):

• A 20% lift on a 2% click rate, to 2.4%, takes about 21,100 people per version. A 10% lift takes about 80,700.
• Doubling a 0.16% order rate, Klaviyo's campaign average, takes about 14,700 per version.
• Opens need fewer people: 40% against 44% takes about 2,400 a version. That's why open tests were popular, and why losing them hurt.
• Chance alone makes winners. Send two identical emails to 500 people each, at a true 2% click rate, and in about two runs out of three one "beats" the other by 20% or more.
• Famous results come from big lists. HubSpot's test of a person's name against the company name in the from line lifted clicks from 0.73% to 0.96%, 292 more clicks. That takes about 127,000 people per version.

To size your own test, a free calculator such as Evan Miller's gives the people needed per version.

On a small list

• Test what could double a result: who gets it, the offer, the main message or angle, short against long, one link against several. A button colour or one word in a subject line can't be measured on a list of a few thousand. The famous red-against-green button test ran on about 2,000 visits.
• Repeat the test. If there were no real difference, the version you expected to win would win five sends out of five only about 3% of the time.
• Add up the sends. Total each version's clicks and deliveries across every send of the same test, then compare the totals.
• Split the whole list rather than testing on a slice and sending the winner. A 10% and 10% test on a list of 5,000 is 500 people a version, which only spots a 2.7-times difference. Mailchimp does the same for small audiences: its test versions "go to all subscribers simultaneously".

Set the finish line first

Evan Miller (2010): "Repeated significance testing always increases the rate of false positives." Check a test again and again, stop when one side leads, and a stated 5% chance of a false winner can become 26.1%.

• Decide how many people, and how long to wait, before the test starts. Clicks keep arriving for a day or more after a send.
• Testing, then sending the winner: a longer wait gives a sounder verdict but sends the winner later. On an offer that ends soon, a short wait can cost more than a wrong pick.

Slow effects: how often, and what email adds

How often to send. Send more often to a random part of the list, keep the rest as it was, and after 8 to 12 weeks compare revenue, replies or bookings per subscriber, and the size of each part. Never judge it by the next email's click rate: more emails always lower the rate per email and usually raise the total: How often to send.

A holdout shows what email adds. Braze recommends holding 1 to 10% of an audience back from all messages, at least 1,000 people and up to 15% for audiences under 10,000, for about three months and never less than one. It answers whether your email adds sales, or only claims sales that would have happened anyway.

• On a small list, 1,000 people can be most of the list. Hold back a random share of new subscribers from one sequence instead, where your tools support it, and read the result once enough people have passed through.
• The held-back people get less, so keep the slice small and the end date fixed.

Sequences: test the top

• The first welcome email reaches every new subscriber and is the best-read email a business sends: 83.63% opens and 16.60% clicks in GetResponse's 2023 data. Test it first.
• A sequence test runs all the time. Each new person gets a version by chance, and you read the result once enough people have been through.
• Test what changes the whole sequence: the first email's offer, the gap between emails, how many there are, the offer at the end. Judge it per person who started the sequence, not per email.

What to test, in order

1. 1

   Who gets it: The right email for the right people.

2. 2

   The offer: what you offer, on what terms.

3. 3

   The message: the angle, and long against short.

4. 4

   The subject line and preview, judged by clicks: From name, subject line and preview.

5. 5

   The button: its words and its place: Get the click.

6. 6

   Design details, last.

Send times and how often you send are worth testing too, but slowly: in the benchmarks, send-time differences are a point or two, and frequency takes months to show.

Check every email before it goes

1. 1

   Every link goes where it should, and none points to a draft or a test page.

2. 2

   The unsubscribe link works.

3. 3

   Every personal field has a fallback, so nobody gets "Hi ,".

4. 4

   The from name, subject line and preview are right, and the preview isn't stray code or "View in browser".

5. 5

   Prices, dates, codes and end dates match your own terms.

6. 6

   Nothing broken shows: no placeholder, no code showing as text, no missing picture, the right logo.

7. 7

   A test send to real mailboxes at the providers your list uses. Apple, Gmail and Outlook make up about 96% of recorded opens (Litmus, September 2026).

The check is for what's broken, not for the words you chose. Each check in full, with how it looks on a phone, in dark mode and with pictures off: Before you press send.

Seed tests send an email to test mailboxes at many providers to see whether it lands in the inbox or in spam. A seed mailbox has never opened your mail, so it shows how a provider treats a stranger, not a subscriber. Use one as a smoke alarm for a sudden move to spam, and your real results to judge the health of your sending: Get into the inbox.

Keep a test log

One line per test: the date, what you tested and why, the deciding measure, the people per version, the result, and what you changed because of it.

A log stops the same test being run twice, shows which results held up when repeated, and turns single wins into rules you can trust. A result from one audience is a starting guess for another, never a law: one test of emoji in subject lines, on 470,000 people, still couldn't tell.

Doing it here

• Every campaign goes out as at least four versions: the same offer, each with a fresh angle and its own subject line, and each person gets one of them: Write your campaign's emails. So every campaign tests its angles and subject lines without extra work.
• The versions differ in angle and subject line together, so the one that leads tells you which whole email did best, not which part made the difference. Note what each version tried.
• Each campaign's report has a Variations tab. It counts every open the way other tools do, beside the opens and clicks by people, and once the versions have had at least 200 sends each, it names the one that leads on clicks by people.
• Read that leader as the best so far. At 200 sends and a 2% click rate, a version has about 4 clicks, and with four versions a list of 2,000 gives each about 500 people, enough to see only a big difference. If the same angle leads again next time, it's a pattern.
• On the Review step, Send Test Email sends every version to the address you choose, each subject line labelled, so you can check them all in a real inbox before you launch.
• Every email's preview switches between phone and computer width, and you can click anything in an email to change it: How your emails are written.
• Every email carries an unsubscribe link in its footer and the one-click unsubscribe headers, and before launch the campaign checks that your brand has a postal address: Unsubscribes, honoured everywhere.
• What stays yours: the test log, and running a test again before you call it a rule.