The screen hasn’t changed from “There isn’t a significant difference yet”. Is there a rough indication of when it will be completed?

• The amount of data that needs to be collected is different to each case, so it is difficult to provide an indication. We will state in general terms hereon after.

• We recommend a two week period for conducting an A/B test since user behaviors are likely to be different on weekdays and weekends.

• The minimum number of delivery count needed to get results for the A/B test is between 100~1,000 per pattern. If the delivery count exceeds more than 1,000 and is not getting results, it is safe to say that there is no difference. However, we can only make suggestions based on the results, and cannot guarantee that there is no difference.

Note

Based on statistics, it is possible to obtain an estimate of necessary delivery counts, from the following parameters.

1. Significance level

2. Power

3. Targeted indicators’ (CVR, open rates etc.) estimated value before the A/B test.

4. The threshold value when determining if the difference has any meaning to it or not

The count of 100~1,000 is the calculated value when the above parameters are given a number under general conditions.

• « At what point is the data collected that is used for computing the significant difference?
• Around how many delivery counts is needed to display the results? »