There are currently no frequently asked questions associated with this exercise – that’s where you come in! You can contribute to this section by offering your own questions, answers, or clarifications on this exercise. Ask or answer a question by clicking reply () below.
If you’ve had an “aha” moment about the concepts, formatting, syntax, or anything else with this exercise, consider sharing those insights! Teaching others and answering their questions is one of the best ways to learn and stay sharp.
Join the Discussion. Help a fellow learner on their journey.
Ask or answer a question about this exercise by clicking reply () below!
Agree with a comment or answer? Like () to up-vote the contribution!
Hi there, when I input tts, a_b_pval = ttest_ind(a, b), the result of the a_b_pval is 2.76676293987e-05, which clearly is not a p-value. What gone wrong?
2.76676293987e-05 is scientific notation for 0.0000276676293987, which is a p-value showing that the result of that particular t-test has an exceptionally good chance at being significant.
What error are we calculating using the error probability function provided? As I understand it, the p-value is essentially the probability of a type 1 error. So, over multiple t-tests:
In this way, multiple t-tests would actually decrease your chance of a type 1 error.
I think my issue here actually boils down to two questions. First, what source of error does the provided error probability function calculate? Second, can p(statistical significance) decrease while p(type 1 error) also decreases?
I think the main confusion here is due to these lessons using the term “p-value” interchangeably for both the significance value (the threshold at which we will determine the results are significant) and the actual p-value that is returned by running a T-test.
Here are the concepts to remember with T-tests:
We are comparing samples of different populations to see if the populations are significantly different
We determine a significance value (or p-value threshold) prior to conducting the T-tests that will act as a cut-off point for whether we will find significance
A T-test returns two values: a test statistic (tstat) and a p-value. The test statistic is basically a number that represents the difference between population means based on the variations in your sample. The larger it is, the less likely a null hypothesis is true. If it is closer to 0, it is more likely there isn’t a significant difference. The p-value is the likelihood of getting a test statistic of equal or higher value to the one returned, if the null-hypothesis is true.
The p-value itself is not the probability of a Type I error, but rather the probability of getting a test statistic (tstat) of equal or higher value if the null hypothesis is true (i.e., if the populations have the same mean and the observed differences were merely by chance). The smaller the p-value, the more likely there is significance.
Prior to running the T-tests, however, we decide that a p-value at .05 or less will indicate significance – thus we are accepting a risk of being wrong 5% of the time when we reject the null hypothesis. We would reject a null hypothesis equally if the p-value was .04 or .00004. Thus, we have a fixed risk of Type I error per T-test that is determined prior to running the experiment. This is the error the lesson is referring to.
This 5% accepted risk is compounded for each T-test we need to run during the experiment to compare each sample with each other sample, and that is why running multiple T-tests can be problematic.
Here’s my take on this. (I’m writing this down to clear my head after a couple of hours in therapy with Dr. Google trying to get it sorted.)
Given these two conditions:
The null hypothesis is true for every test we run.
We agree to reject the null hypothesis if the t-test returns a p < 0.05
The question is:
If I do n comparisons via a t-test, what is the probability that I will see at least one result out of that series that tells me that the null hypothesis is false (i.e., at least one false positive)?
We can agree that if n = 1, the probability is 5%, or 0.05. The probability that I will be correct, i.e., will not make the mistake of publishing a false positive, is 0.95
We’re rolling a 20-sided die here. One side says “reject”, the other 19 say “accept”.
If I roll twice, what is the probability of at least one “accept”?
The best way to look at this is to realize that the probability of getting at least one hit in n tries is the inverse (1 - p) of the probability of never getting a hit in n tries.
So, what is the probability of never getting an “accept” in 2 tries: it is P(reject) * P(reject) = 0.95**2, and the inverse of that is (1 - 0.95**2), or 0.0975, or nearly 10% chance of one “reject” in two rolls.
Likewise, for any number, n, of rolls, the probability of never getting “accept”, i.e., getting “reject” on every roll, is P(reject) **n, which in our specific case is 0.95**n.
As above, if the probability of never getting any “accept” is 0.95**n, the probability getting at least one “accept” is (1 - 0.95**n), or 0.142625 for n = 3.
So, what the problem is saying is, "If the three samples are in fact not closely correlated (i.e., the null hypothesis is true), then there is a 14.26% probability that you will see at least one “reject the null hypothesis” condition if you run the t-test three times."
It seems the 0.05 pval that the lesson is asking you to assume is an arbitrary number when the actual pvals are calculated in the lesson. Why wouldn’t we use the actual pval of 2.76676293987e-05 +
0.0210120516986 + 0.0598856352397 ?
Awesome explanation, thanks!
I have another question tho, if the error probability gets bigger with the more t-tests we do, is it possible to select a higher p-value threshold (significance) so the compounded error probability is still acceptable?
To put it in the context of this exercise:
1-0.95**3 = 0.142625
1-0.97**3 = 0.087327 (which is higher than 0.05, but still kind of acceptable)
1-0.99**3 = 0.029701 (which is lower than 0.05)
My first guess is that this would be a problem or at least an inconvenience of some sort, but I’m still trying to wrap my head around this
The idea is exactly what is used in some standard ways to avoid the dangers described in this exercise. A simple one is Bonferroni correction, which keeps the overall significance level low by lowering the significance level of each test (by dividing by the number of hypotheses). You’ll learn another way called Tukey’s range test in a later exercise. This is a more complicated approach, but it is based on a similar idea.
) below.
) to up-vote the contribution!