An A/A test is a test where the variant group is the same as the control group.
It may seem silly to test two identical groups, but the goal of an A/A test isn't to test a change to one of the groups — the goal is to evaluate the statistical rigor of the tool used to run experiments.
You can set up an A/A test in Leanplum by creating an A/B test, then building an identical Control and Variant(s).
No two audience groups will be exactly alike. Even with good experiment design (sufficiently large sample sizes, random selection, and user assignment), the metrics for each test group will likely show some differences.
If many metrics show significant differences, it's possible there's a statistical problem with the tool's experimentation methodology. If you are experiencing these types of results in an A/A test, contact your CSM.
Updated over 1 year ago