Gradually rolling out A/B tests to your audience

The rate at which you roll out your A/B tests to new users should depend on the size of your total audience as well as how fast you want to get results. Most Leanplum users implement a two-to-three stage rollout process, starting with a small portion of their total audience and slowly increasing the sample size to a higher percentage of the audience until finally reaching 100%. This type of staged roll out plan can help minimize the risks associated with testing a new feature.

Starting the test with a smaller audience

The Audience Slider is the best way to gradually roll out your A/B tests. Audience represents the percentage of targeted users that should be included in the experiment. Included users are randomly assigned to either the control or one of the variants.

🚧

If you are testing a message, make sure that message is paused when using the audience slider. A live message may be delivered to users targeted outside of your test, including those outside the audience selected with the slider.

Also note that the audience slider can not be used with immediate messages. See A/B testing a message for more details on these settings.

By default, 100% of users who meet your targeting requirement are included in the A/B test. If you want to roll out starting with a smaller audience, you should reduce the percentage prior to starting the test.

Notice that when we reduce the audience size to 1%, the Excluded group appears next to Variant 1 in the test table below. This represents the percentage of targeted users that are excluded from the experiment.

❗️

Audience slider note

Once you publish your test, you cannot reduce the audience size — you can only increase it. This is because targeting changes only affect new users, not users who have already been assigned to a test group. Make sure you start with a smaller audience before publishing your test for the first time!

Increasing the size of the audience step-by-step

If you are ready to increase the sample size in your experiment, you can roll it out to a wider audience. In this example, we increase the audience from 1% to 50%, and then again from 50% all the way up to 100%.

Notice that each time we increase the audience, we do not modify the distribution, the rate at which users are assigned to each variant. It is best practice NOT to modify the distribution among variants mid-test, as doing so can bias your variant groups and make it difficult to interpret test results.

Choosing a winner and finishing the test

If you have rolled out to 100% of the audience and are ready to pick a winning variant, we recommend that you finish your test and roll out the winning variant to the desired group of users.

Please reach out to [email protected] or to your customer success manager with any questions.