What is the difference between the _p-value_ and the _significance level_ of an experiment?

Question

What is the difference between the p-value and the significance level of an experiment?

Answer

The p-value is a statistic that we can compute as a conditional probability. It is the probability that we would observe the same sample statistics given that the null hypothesis, H_0, is true. As an equation:

p-value: P( observe the same sample statistics | H_0)

By way of an example, let’s say we’re performing a test and we want to determine if there is a significant increase in time spent on our website if we change the background color from white to yellow. Suppose that the current average time spent on our website is 15 minutes. After changing the background color to yellow, we take a sample of 100 users and we find that the average time spent on the site is 20 minutes. Does this result show a significant change or not? The significance level, produced before we perform the sample, is the threshold value from which we will determine to reject, or fail to reject, the null hypothesis; often chosen as 0.05. The p-value, however, is the following conditional probability,

P( sample mean >= 20 minutes | background of site is unchanged )

The sample mean is the average time spent on the site from a given sample and the background of the site being unchanged is the null hypothesis. To conclude, the significance level is a threshold value chosen beforehand and the p-value is a conditional probability to calculate the significance of an observation.

3 Likes

I think the explanation of this FAQ is exact and the answer of the following quiz is incorrect so it should be fixed:


The third choice is not a correct description of a p-value, so none of three is correct.

2 Likes

This article might be helpful:
https://doi.org/10.1080/00031305.2016.1154108
See the second principle:

  1. P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.

I’ll leave this here, I found it really helpful to understand the p-value concept:

Cheers :beer:

6 Likes