A/B Testing Your Emails: A Simple Guide for Small Businesses

How to A/B test email subject lines, content and send times — what to test, how big your sample should be, and how to read the results.

In short: Test one change at a time on a list large enough to give a clear answer, judge the winner by clicks or sales rather than opens, and keep a simple log so each test teaches you something for the next email.

A/B testing means sending two versions of an email to small groups of your list, seeing which performs better, and sending the winner to everyone else. It replaces "I think" with "we know" — and you don't need to be a data scientist to do it.

How it works

Pick one thing to change (for example, the subject line).
Create version A and version B.
Send each to a random slice of your list — say 15% each.
Wait a few hours (or a day).
Send the winning version to the remaining 70%.

What to test (in order of impact)

Subject line — the biggest effect on whether your email gets read.
Offer — "20% off" vs "free shipping" vs "free gift".
Call to action — the button's wording and position.
Send time — morning vs evening, weekday vs weekend.
Content length — short and punchy vs detailed.
Sender name — "Business name" vs "Person at Business name".

Test one thing at a time

If you change the subject line and the offer, you won't know which change made the difference. Keep everything else identical.

Choose the right winning measure

Testing subject lines? Opens are a rough guide, but clicks are more reliable since Apple's privacy changes inflated opens.
Testing content or offers? Use clicks or, better, conversions.

How big does my list need to be?

Small lists can still test, but small samples give noisy results. As a rough guide, you want at least a few hundred recipients per version before trusting a difference — and the smaller the difference, the more people you need. If your list is small:

Test big, bold differences (not "Shop now" vs "Shop today").
Repeat a test across several campaigns before drawing conclusions.
Use a 50/50 split of the whole list and learn for next time.

Read results honestly

A tiny difference between versions is often just chance.
Look for consistent winners across several tests.
Write down what you learned ("Questions beat statements in our subject lines") and use it.

Ideas for your first five tests

Subject line with a number vs without.
Including the subscriber's first name in the subject vs not.
Discount vs free shipping.
One button vs three links.
Morning vs evening send.

A worked example

A boutique wants to know whether a discount or free shipping drives more sales for its spring launch.

Version A: "Spring is here — 15% off the new collection".
Version B: "Spring is here — free shipping on the new collection".
Each version goes to 20% of the list; the winner metric is conversion rate (orders), measured after 24 hours.
Free shipping wins with more orders, so the remaining 60% receive version B.
The boutique repeats the test at its summer launch to confirm the lesson before making it the default.

Keep a testing log

A simple spreadsheet is enough: date, what you tested, the versions, the winner, the difference and whether it was significant. After a few months you'll have a playbook of what works for your audience — far more valuable than any generic best-practice list.

Mistakes to avoid

Declaring a winner after an hour.
Testing tiny changes on a small list.
Running a test during an unusual week (a holiday, a big news event) and treating the result as permanent.
Forgetting to apply what you learned.

Understanding results without the math

A/B tests can mislead you if you don't account for chance. Here's the intuition.

Imagine you send version A and version B to 100 people each. A gets 6 clicks, B gets 4. Is A better? Probably not provably — with such small numbers, a couple of people clicking or not could easily flip the result. Now imagine 2,000 people per version: 120 clicks versus 80. That difference is much more likely to be real.

Rules of thumb for small businesses:

Bigger differences need fewer people. A subject line that doubles clicks shows up quickly; one that improves them by 5% needs thousands of recipients to detect.
Look at the number of clicks, not just the percentage. Results based on a handful of clicks are mostly noise.
Repeat tests that matter. If emoji subject lines win three times in a row, you can trust the pattern far more than one result.
Don't stop a test early because one version is ahead after an hour. Early results are often reversed.

When not to A/B test

Very small lists (a few hundred people): you'll rarely get a clear answer. Focus on writing good emails and learn from replies and overall trends instead.
One-off, time-critical emails like a same-day closure notice.
When you already know the answer from previous tests.

For small lists, test across several campaigns instead: use one style for a month, the other the next month, and compare average clicks.

In Reepio

Create an A/B test on any campaign, choose what to test and how to pick the winner (opens, clicks, click-to-open, conversions or revenue), and Reepio sends the better version to the rest of your list automatically — and records whether the difference was statistically significant, so you know which lessons to trust. Start free.

Frequently asked questions

How long should I wait before picking a winner?

At least a few hours, and ideally a full day for clicks and conversions. Waiting too little favours people who check email quickly.

What should I test first?

Subject lines, because they have the biggest effect on whether your email is read. Then test offers, calls to action and send times.

Can I A/B test with a small list?

Yes, but test bold differences and repeat tests over several campaigns, because small samples make results noisy.