The screen hasn’t changed from “There isn’t a significant difference yet”. Is there a rough indication of when it will be completed?¶
The amount of data that needs to be collected is different to each case, so it is difficult to provide an indication. We will state in general terms hereon after.
We recommend a two week period for conducting an A/B test since user behaviors are likely to be different on weekdays and weekends.
Significance testing requires a delivery count (sample size) of at least 100 per variant.
Repro’s significance testing uses a method (sequential testing) designed so that the probability of error does not increase even when you check the results repeatedly. Because of this property, it tends to require more data before reaching a decision than methods that run a test only once.
If no significant difference appears despite a sufficiently large delivery count, you should assume there is no meaningful difference between the variants.