What is the difference between the p-value and the significance level of an experiment?
The p-value is a statistic that we can compute as a conditional probability. It is the probability that we would observe the same sample statistics given that the null hypothesis,
H_0, is true. As an equation:
p-value: P( observe the same sample statistics | H_0)
By way of an example, let’s say we’re performing a test and we want to determine if there is a significant increase in time spent on our website if we change the background color from white to yellow. Suppose that the current average time spent on our website is 15 minutes. After changing the background color to yellow, we take a sample of 100 users and we find that the average time spent on the site is 20 minutes. Does this result show a significant change or not? The significance level, produced before we perform the sample, is the threshold value from which we will determine to reject, or fail to reject, the null hypothesis; often chosen as
0.05. The p-value, however, is the following conditional probability,
P( sample mean >= 20 minutes | background of site is unchanged )
The sample mean is the average time spent on the site from a given sample and the background of the site being unchanged is the null hypothesis. To conclude, the significance level is a threshold value chosen beforehand and the p-value is a conditional probability to calculate the significance of an observation.