What is a significant difference?¶
It is a statistical judgment of whether the difference in the target indicator (such as CVR, open rate, or button tap rate) is not due to chance. The calculation uses Always Valid Confidence Sequences, a sequential testing method designed so that the probability of a false judgment does not increase even with repeated evaluations. For details, see for example Statistical inference.