FAQ: Hypothesis Testing - Dangers of Multiple T-Tests


#1

This community-built FAQ covers the “Dangers of Multiple T-Tests” exercise from the lesson “Hypothesis Testing”.

Paths and Courses
This exercise can be found in the following Codecademy content:

Data Science

FAQs on the exercise Dangers of Multiple T-Tests

There are currently no frequently asked questions associated with this exercise – that’s where you come in! You can contribute to this section by offering your own questions, answers, or clarifications on this exercise. Ask or answer a question by clicking reply (reply) below.

If you’ve had an “aha” moment about the concepts, formatting, syntax, or anything else with this exercise, consider sharing those insights! Teaching others and answering their questions is one of the best ways to learn and stay sharp.

Join the Discussion. Help a fellow learner on their journey.

Ask or answer a question about this exercise by clicking reply (reply) below!

Agree with a comment or answer? Like (like) to up-vote the contribution!

Need broader help or resources? Head here.

Looking for motivation to keep learning? Join our wider discussions.

Learn more about how to use this guide.

Found a bug? Report it!

Have a question about your account or billing? Reach out to our customer support team!

None of the above? Find out where to ask other questions here!


#2

Hi there, when I input tts, a_b_pval = ttest_ind(a, b), the result of the a_b_pval is 2.76676293987e-05, which clearly is not a p-value. What gone wrong?


#3

2.76676293987e-05 is scientific notation for 0.0000276676293987, which is a p-value showing that the result of that particular t-test has an exceptionally good chance at being significant.


#4

What error are we calculating using the error probability function provided? As I understand it, the p-value is essentially the probability of a type 1 error. So, over multiple t-tests:

p(type 1 error) = p_value_1 * p_value 2 * … * p_value_n

thus:

p(not(type 1 error)) = 1 - (p_value_1 * p_value 2 * … * p_value_n).

In this way, multiple t-tests would actually decrease your chance of a type 1 error.

I think my issue here actually boils down to two questions. First, what source of error does the provided error probability function calculate? Second, can p(statistical significance) decrease while p(type 1 error) also decreases?