Adding a Holdback to your message

📘

Holdbacks require Leanplum SDK version iOS 1.4.1 / Android 1.3.1 or later.

At Leanplum, we believe every piece of content should be tested. That's why we make it easy to A/B test messages and see how successful they are at driving user behavior and ROI.

By withholding your message from a group of users, you can measure how successful your message is at driving behavior compared to the holdback group. Using the holdback feature in conjunction with a Message goals can give an even clearer picture of the user behavior generated by your message.

Adding a Holdback

To add a holdback group to your message, click Add Holdback in the upper right corner of the Target.

By default, 10% of targeted users will be held back from receiving the message. To change the percentage, click the 10% and type in a new percentage.

Users who are assigned to the holdback will not receive the message — no need to set up a separate A/B test. We handle all of that behind the scenes! Leanplum will create a control group that doesn't receive the message and compare results to the test group that does.

📘

"Held back" event tracking

Users who don't receive your message as a result of a holdback will track a "held back" event in Analytics and their user profile. This event means the user would have got the message if no holdback was set. See Holdback results for more info.

You can also change the holdback in the advanced holdback settings.

1. Click Add Holdback, then click the gear icon. Select View Holdback Test to open your advanced settings.

2. To change the holdback percentage for your message, use the Distribution slider to adjust the percentage of users in the control group.

The control users are the holdback group, and they will not receive your message. (The Enabled checkbox for the control group should stay unchecked.)

You can also use the holdback to send a message to a smaller portion of users. For example, to send your message to just 20% of your audience, move the control group's distribution (the holdback group) up to 80%.

Analyzing the results

Leanplum will track a few Events for users targeted in a message with a Holdback. Everyone who qualifies for the message's target and delivery settings will track a Holdback impression event in Analytics and in their user profiles, whether or not they receive the message.

The users who receive the message will also track a message Sent event in Analytics and in their user profiles. The users who do not receive the message (because of the holdback) will trigger a Held back event at the time the message would have been sent.

All these events will be visible in the user profiles in the Users tab, as well as in the Analytics for the message itself. See Events tracked by default for a full list with descriptions of Leanplum's default events.

📘

For immediate and scheduled messages, we will automatically finish the Holdback test two days after the last message occurrence. For triggered messages, we will automatically finish the test 30 days after you finish the message.

Advanced uses

You can also use Holdbacks for more advanced message use cases.

We recommend adding a Holdback group to all of your messages to best determine the impact of your messages.

Two different message copy versions
You can easily add additional variants to the Holdback if you wish to test multiple versions of copy, images or other message content.

1. Click Add Holdback in your message.
2. Click the gear icon next to Holdback to view the Holdback Test.
3. Add another variant to the test and make any desired message copy changes.
4. Start your message.

Note: Holdbacks can't be added to previously-active messages. To add a Holdback to an existing message, please create a new copy of your message.

We recommend using Holdback in conjunction with Message goals ls to help you understand the full impact that your message campaigns are having on users.