Glossary / Statistical significance
What is Statistical significance?
Statistical significance is the test of whether a difference between two results is real, or small enough that luck could explain it.
Version A converts at 4 percent, version B at 5 percent. Significance testing asks a plain question. If the two were actually identical, how often would chance alone produce a gap this big? If the answer is rarely, the result is called significant.
Sample size decides it. With 50 visitors per version, almost nothing is significant, because a handful of conversions swings the whole percentage. Small tests produce confident-looking numbers that mean nothing.
Two habits protect you. Set the sample size and the end date before you start, and do not stop the moment the result looks good. Peeking and stopping early is the fastest way to crown a winner that never repeats.
In Creaiter
Creaiter's chart analysis describes what your numbers did in plain language, which helps you notice when a swing is too small to act on.
Related terms