FAQ: Analyze FarmBurg's A/B Test - Performing a Significance Test II

This community-built FAQ covers the “Performing a Significance Test II” exercise from the lesson “Analyze FarmBurg’s A/B Test”.

Paths and Courses
This exercise can be found in the following Codecademy content:

Data Science
Analyze Data with Python

FAQs on the exercise Performing a Significance Test II

There are currently no frequently asked questions associated with this exercise – that’s where you come in! You can contribute to this section by offering your own questions, answers, or clarifications on this exercise. Ask or answer a question by clicking reply (reply) below.

If you’ve had an “aha” moment about the concepts, formatting, syntax, or anything else with this exercise, consider sharing those insights! Teaching others and answering their questions is one of the best ways to learn and stay sharp.

Join the Discussion. Help a fellow learner on their journey.

Ask or answer a question about this exercise by clicking reply (reply) below!
You can also find further discussion and get answers to your questions over in Language Help.

Agree with a comment or answer? Like (like) to up-vote the contribution!

Need broader help or resources? Head to Language Help and Tips and Resources. If you are wanting feedback or inspiration for a project, check out Projects.

Looking for motivation to keep learning? Join our wider discussions in Community

Learn more about how to use this guide.

Found a bug? Report it online, or post in Bug Reporting

Have a question about your account or billing? Reach out to our customer support team!

None of the above? Find out where to ask other questions here!

Hello,

I´ve done the exercise and ther something I don´t understand

We accept the hypotesis null when the price is 4.99

But if you try a random binomial (like: sum(np.random.binomial(1666,83/1583,20000)>=201)/20000) the probability is 0, and it makes sense because you need 201 success when the average success is 83.

How can be posible that the binomial test accept the null hypotesis?

Thanks and regards

Pablo Arias

Hello,

I’ve been stuck in the second test for so long I decided to click on the solutions button. Yet I still don’t understand. Where does the number 316 and 1666 come from?

If x = the number of purchases for Group A, shouldn’t it be 1011 and not 316?
If n = the total number of visitors assigned Group A, shouldn’t it be 4998 and not 1666?
Is there anything I’m missing here?

EDIT: Finally figured it out!

316 came from the ‘Group A is_purchase’ row in the dataset provided at the top of the exercise.
1666 came from either 4998 / 3 OR 316 + 1350.

Thanks :smiley:

Hello,

After the exercise, we know that there is a significant difference between the observed purchase rate and the expected one.
But we CANNOT know that the observed purchase is significantly greater or significantly less than the target percent of purchases.
So, how can we say that the price of $4.99 is the best one we should choose?

Can anyone explain it to me?

Thank you so much.
Best regards./.

3 Likes

Hi Igk1910

hope this helps you

We already calculated how many percent we should have, which is approx. 4%

p_clicks_499 = (1000 / 4.99) / num_visits

If we could achieve this conversion rate or above, that means we would be successful
we knew that 83 out of 1666 is 4.98% so want to make sure whether this is fluctuation or significant difference.
In the exercise, we got p-value of 0.045623 so concluded that significantly greater difference

If you simulate this case with using numpy (np.random.binomial(83, 1666, size=10000)), you will be able to see the distribution.
Here’s attached a binomial graph I made.

test
※Blue vertical line indicate x=83, which is number of people who purchased the product

4 Likes

Oh, I forgot that there was a ratio of 83 over 1666. Thanks for your help :hugs: !!!

Hi lgk1910,

I had the same issue with the way the problem was stated. They don’t point out anywhere that in this case study, there’s really two criteria that we care about when looking at the results of each binomial test for groups A, B, C:

  • p-value < 0.05
  • actual purchase rate > target purchase rate

Their sample data conveniently works out in a way that ONLY group C has a significant p-value, and it just so happens to meet both those criteria. This would get super confusing if the data was a little different though, like if you tweaked the sample data so that group A had 300 purchases instead of 316. You’d get a p-value of 0.04 that suggested “actual purchase rate of Group A is significantly different than target purchase rate of Group A”, but it wouldn’t tell you that it’s actually significantly LOWER, meaning we definitely don’t want to go with this option.

By checking both the p-value and whether (Actual Purchase Rate) > (Target Purchase Rate), you can know for sure which is the best option. I wish they’d emphasized that in the lesson.

6 Likes

I have a couple of questions. It looks like $1.99 would also allow VaryAnts to make their $1000 sales goal. Why is the pval for Group B so high?

Also - What if we were to make significantly more money than we needed? Would this still be a good test if the success rate of Group C were 10%? 10% is further away from the expected success rate of 4%, but it would be a preferable outcome.

Note: Codecademy would not calculate fractions for some reason, so I have to create this on my Jupyter server. Code and output below

import pandas as pd
from scipy.stats import binom_test

df = pd.read_csv(‘clicks.csv’)

num_visits = len(df)
#percentage of visitors who would need to buy to make the $1000 #a week goal
p_clicks_099 = (1000 / 0.99) / num_visits
p_clicks_199 = (1000 / 1.99) / num_visits
p_clicks_499 = (1000 / 4.99) / num_visits

print(df.head())

pvalueA = binom_test(316, 1666, p_clicks_099)

pvalueB = binom_test(183, 1666, p_clicks_199)

pvalueC = binom_test(83, 1666, p_clicks_499)

final_answer = 4.99

print(’\n’)
print(‘Number of Sales needed for A, B, C to meet $1000 weekly breakeven point’)
print(1000 / 4.99)
print(1000 / 1.99)
print(1000 / 0.99)
print(’\n’)
print(‘Total Visits - ’ + str(num_visits))
print(’\n’)
print(‘Percentage of Sales Needed for sales Goal’)
print(p_clicks_099)
print(p_clicks_199)
print(p_clicks_499)
print(’\n’)
print(‘Actual Success Rates for A, B, C’)
print(316 / 1666)
print(183 / 1666)
print(83 / 1666)
print(’\n’)
print(‘Pvalues for A, B, C’)
print(pvalueA)
print(pvalueB)
print(pvalueC)

#Output from code above Below this line

Number of Sales needed for A, B, C to meet $1000 weekly breakeven point
200.40080160320642
502.51256281407035
1010.1010101010102

Total Visits - 4998

Percentage of Sales Needed for sales Goal
0.20210104243717691
0.10054272965467594
0.040096198800161346

Actual Success Rates for A, B, C
0.18967587034813926
0.10984393757503001
0.04981992797118848

Pvalues for A, B, C
0.2111287299402726
0.20660209246555486
0.045623672477172125

I have a question about the way we calculate p_clicks_x99 variables. We assume that num_visits are the dataframe length. Shouldn’t we have to calculate the percent of clicks on each group? Shouldn’t we have to divide by the number of visitors of each group?

Thank you. If figured out the code, I could execute the test and produce the A,B,C values, but could not say, didn’t fully understand which choice to make. Your explanation is very much appreciated

How come we tested Group A with .99, Group B with 1.99 and Group C with 4.99?

Could we have tested A with 4.99? B with .99? C with 1.99? or some other combination?

Here’s how I got to the solution on the last step:

import codecademylib
import pandas as pd
from scipy.stats import binom_test

df = pd.read_csv('clicks.csv')
df['is_purchase'] = df.click_day.apply(lambda x: 'Purchase' if pd.notnull(x) else 'No Purchase')
num_visits = len(df)

p_clicks_099 = (1000 / 0.99) / num_visits
p_clicks_199 = (1000 / 1.99) / num_visits
p_clicks_499 = (1000 / 4.99) / num_visits
print(df.head())



xa = df[(df['group'] == 'A') & (df['is_purchase'] == 'Purchase')]
na = df[ df['group'] == 'A']

pvalueA = binom_test(len(xa), len(na), p_clicks_099)

xb = df[(df['group'] == 'B') & (df['is_purchase'] == 'Purchase')]
nb = df[ df['group'] == 'B']

pvalueB = binom_test(len(xb), len(nb), p_clicks_199)

xc = df[(df['group'] == 'C') & (df['is_purchase'] == 'Purchase')]
nc = df[ df['group'] == 'C']

pvalueC = binom_test(len(xc), len(nc), p_clicks_499)

print(pvalueA)
print(pvalueB)
print(pvalueC)

final_answer = 4.99

I’m finding Data Analysis quite hard to grasp, looking at the solution, it felt so much easier just to generate a table that counts each instance…
It feels like a cheatsheet would help a ton!

So, this is badly worded and explained.
We are using the binomial test to see if the actual proportion of customers who purchased at each price point is significantly different to the proportion that we need to meet out target per week of 1000 dollars from the total weekly visitors (4998 of them).
A and B are significantly different; C is not significantly different - and presumably that makes it our best bet for achieving our target per week.
I think the problem I had with this is that ‘significantly different’ could be much more than our target as well as much less than our target. If A or B made us much more than our target, they would be preferable.
So do the maths.
A: assume 18.9% (316/1666) of our 4998 customers per week purchase at 0.99. That is 935.17, some 64 dollars under and is significantly different to our target of 1000.
B: assume 10.9% (183/1666) of our 4998 customers per week purchase at 1.99. That is 1084.11, some 84 dollars over and is significantly different to our target of 1000.
C: assume 4% (83/1666) of our 4998 customers per week purchase at 4.99. That is 997.30, less than 3 dollars under, and is NOT significantly different to our target of 1000.

So C is closest to our 1000 dollar target, but B actually makes us MORE than our target. So - why not B?
Because the binom pvalue doesn’t tell us if the significant difference is bigger than or lower than our target, we have to do a little more work, I think, than the lesson tells us to do. C would be ‘safe’ and we would cover the costs, but B would make us more profit.

3 Likes

In the calculation of the percentages, I think you’ve rounded numbers differently between A, B, and C. Especially for C, 83/1666 should actually be pretty close to 5%, but unlike A and B, you’ve truncated the decimal point and rounded it to 4%.

(316/1666) * 4998 * 0.99 = 938.5199…
(183/1666) * 4998 * 1.99 = 1092.5099…
(83/1666) * 4998 * 4.99 = 1242.5099…

2 Likes

I completely agree with you! My thought was that the 1.99 price point produced a significantly greater outcome than the target, meaning it made the company more money. I’ll have to brush up on my Stats knowledge and vocabulary to understand why Codecademy chose 4.99.