An A/B test sends versions of a campaign email or push notification to part of its audience, split evenly. After the test, the winner goes to the rest of the audience: the version with the highest rate of the measure you chose, picked automatically, or the version you choose. You can also stop without sending to anyone else. For email, test the subject, the preview text, the sender name or the whole design; for push, the title, the message or the image.
Set up a test
- Create an email campaign and write its email. See Create a campaign.
- On the Email step, under A/B test, tick Test versions of this email. The email you wrote is version 1.
- Under Version 2, change what this version tests: Subject, Preview text, From name, or choose Use another design. Anything you leave blank is version 1’s. To test only the subject, change just the subject.
- To test more, choose Add version 3, then Add version 4. A campaign can have up to four versions.
The sender’s address is the same for every version; only the From name can change.
Test a push notification
A campaign that sends push only can test its notification instead. On the Push notification step, under A/B test, tick Test versions of this notification, then change the Title, Message or Image URL for each version. Web and Android show the image; iOS needs the app’s service extension.
A test covers one channel: a campaign that sends both email and push can’t have one.
Choose how the winner is picked
- Test on: the share of the audience the versions go to, from 10% to 50%, split evenly between them. The rest get the winner. It’s 20% unless you change it.
- Winner: for email, Most clicks, Most opens or Most conversions; for push, Most taps or Most conversions. Each is compared as a rate, such as the click rate, so a version that happened to reach a few more people doesn’t win on numbers alone. A tap counts only when the notification opens a link, so add one on the Push notification step or pick the winner by conversions. For conversions, available on Starter and above, choose one of the campaign’s conversion goals; add the goal on the campaign’s page first. See Conversion goals.
- Pick the winner: Automatically after 1 to 72 hours (4 unless you change it), or I’ll pick it myself.
Versions are compared on email opens and clicks as a share of each version’s delivered messages, as on the campaign’s report, or as a share of sent until any version has a delivery. Push taps are a share of each version’s sent messages, because push has no delivered count, and conversions are the share of each version’s sent messages that reached the goal. If two versions tie, the first of them wins.
Before you send, Who receives it warns when each version would reach fewer than 1,000 contacts. Small samples may not produce a clear winner, so test on a larger share or with fewer versions. When fewer than one contact is expected per version, a stronger warning says a version may reach nobody. Neither warning stops the send. See Warnings before you send.
Send the test in each contact’s local time
On the schedule, choose At this time in each contact’s time zone. The test versions go out at that time where each contact is, and the rest get the winner at the same local time when it next comes round, usually the next day. If you choose Send at that time tomorrow for contacts where the time has already passed, the winner is picked about a day later than usual, once they’ve had the test too. See Schedule and send a campaign.
Send the test
Review and send the campaign as usual; the review shows the test, for example 2 versions on 20% of the audience · most clicks after 4 hours, and the split to expect among the contacts who can receive it, such as About 1,000 per version; about 8,000 get the winner. Each contact is drawn at random, so the real numbers vary around these. See Schedule and send a campaign.
While the test runs, the campaign’s status is Testing. To send a version to the rest yourself, choose Send to the rest on its row and confirm. You can do this before an automatic pick, too. Choose Don’t send the rest to stop the test without sending to anyone else.
Sending the winner follows the same rule as sending a campaign: if your workspace requires an admin to send, only admins and the owner can choose it. Kiro AI and other agents can set a test up, but a person sends it and picks the winner.
Read the results
The campaign’s page shows an A/B test panel with each version’s Sent, Opened by people and Clicked (for push, Sent and Tapped), and Converted when conversions decide. Percentages are worked out the same way as when the winner is picked (see above). The winner is marked Winner. The panel covers only the people in the test; the campaign’s totals include everyone.
Use Preview with the Version menu to see each version as a contact would.
Limits
- A test sends email only or push only. A campaign that sends both can’t have one, and journeys and messages your app sends through the API can’t be tested.
- Kirora picks the version with the highest rate of your chosen measure. It doesn’t test whether the difference is large enough to be more than chance, so give each version enough people.