What is Significance?
In an AB test, statistical significance indicates how likely it is that the difference between the control version and the test version of your experiment is not due to error or chance.
For example, if you run a test with a 95% confidence interval, you can be 95% sure that the differences are real.
Companies often use this to see how experiments affect their conversion rates. In surveys, statistical significance is usually used to determine the reliability of survey results. For example, in a survey, you have asked people if they prefer ad concept A or ad concept B. Datatrics also uses this in its A/B testing. You then want to be sure that the difference in the results is statistically significant before you decide which concept to use.
How to calculate the p-value significance?
These are the steps for calculating the p-value on paper:
Determine the expected results for your experiment.
Calculate and determine the observed results of your experiment.
Determine the degree of freedom - how much deviation from the respected results counts as significant?
Compare the initial, expected results with the observer's results using a chi-square.
Choose the significance level (this is where .05 is usually used).
Approach your p-value using the chi-square distribution table.
Reject or retain your null hypothesis.
As you can see, there is quite a bit involved and to take into account when doing this with pen and paper. You need to check that you have followed the correct formulas for all the steps and also check again that you have the correct values.
To avoid the risk of getting wrong results due to bad calculations, it is best to use tools like Google Spreadsheets or Microsoft Excel. since the p-value is so important, the developers have added a feature that calculates it directly. You can also use an online calculator.
In Datatrics
When Datatrics displays "Significant", it means that Datatrics determined using calculations that this touchpoint is performing well without any coincidence. Datatrics uses the p-value of max. 0.05 to assign the Significant label.
Click here to view this image on a larger scale.
In this example above, 2 out of 4 touchpoints are Significant. This might be strange to you, as the conversion uplift of the Persuasion General is higher (58%) than Payment methods - Inverse (28%).
By using the “Export to CSV” button on the Reporting page, you can retrieve the statistics from the touchpoints above. By using the Significance function on this data you will see the following:
Click here to view this image on a larger scale.
As you can see, the second and fourth touchpoints have p-values below 0.05. This is why those touchpoints have the Significant label.
Important note: Datatrics uses the Unique shows, and Unique hidden to calculate conversion rates as one person can see a touchpoint multiple times in some cases.
Datatrics has as thresholds to assign significance at least ten conversions in both groups and at least 100 unique impressions