Repro - Mobile Analytics for growth
日本語
Sign Up Back to Dashboard
  • System Requirements
  • Dashboard Guide
  • Development Guide
  • Release Notes
  • FAQ
    • App
    • Web
    • General
      • Analytics
      • A/B test significant difference display
        • At what point is the data collected that is used for computing the significant difference?
        • The screen hasn’t changed from “There isn’t a significant difference yet”. Is there a rough indication of when it will be completed?
        • Around how many delivery counts is needed to display the results?
        • What is a significant difference?
        • What is a confidence interval?
        • Will the results change if it was determined to have a significant difference?
        • Is it possible to go back to past significant differences?
        • Is it possible to change the significance level used in the confidence interval?
        • When is automatic winner selection performed?
        • Can I enable automatic winner selection for a campaign that is already being delivered?
        • Can the control group be automatically selected as the winning variant?
      • API
      • Miscellaneous

The screen hasn’t changed from “There isn’t a significant difference yet”. Is there a rough indication of when it will be completed?¶

  • The amount of data that needs to be collected is different to each case, so it is difficult to provide an indication. We will state in general terms hereon after.

  • We recommend a two week period for conducting an A/B test since user behaviors are likely to be different on weekdays and weekends.

  • Significance testing requires a delivery count (sample size) of at least 100 per variant.

  • Repro’s significance testing uses a method (sequential testing) designed so that the probability of error does not increase even when you check the results repeatedly. Because of this property, it tends to require more data before reaching a decision than methods that run a test only once.

  • If no significant difference appears despite a sufficiently large delivery count, you should assume there is no meaningful difference between the variants.

  • « At what point is the data collected that is used for computing the significant difference?
  • Around how many delivery counts is needed to display the results? »

Repro Service Site About Us Careers Terms of Service Privacy Policy Cookie Policy

© 2022 Repro Inc.