Will the results change if it was determined to have a significant difference?¶
It may change until a winning variant is selected. The results are recalculated each time they are displayed, using all delivery results up to that point, so depending on subsequent delivery results, “significant difference” may revert to “no significant difference”. However, Repro’s significance testing uses a method (sequential testing) that rarely produces a false “significant difference” even with repeated checks, so a displayed result is unlikely to be overturned later.
Once a winning variant is selected (either manually or by automatic selection), no further significance testing is performed, so the results no longer change. If “Automatically select the winning pattern once a significant difference is confirmed” is enabled, the results at the time of the decision are displayed as fixed, so there is no concern that the judgment will change afterwards.