This project was fun to do. I worked my way through before checking the solution so most of my code will differ from the solution. My question filtering functions for steps 2 and 3 also exclude substrings which is why they return a smaller dataset than the solution. Does my code need to be cleaner? Are there any obvious downsides to the way coded this project? Any feedback would be appreciated.
-----------------------------------------------------------------------------------------------------------
-
We’ve provided a csv file containing data about the game show Jeopardy! in a file named
jeopardy.csv. Load the data into a DataFrame and investigate its contents. Try to print out specific columns.Note that in order to make this project as “real-world” as possible, we haven’t modified the data at all - we’re giving it to you exactly how we found it. As a result, this data isn’t as “clean” as the datasets you normally find on Codecademy. More specifically, there’s something odd about the column names. After you figure out the problem with the column names, you may want to rename them to make your life easier for the rest of the project.
import pandas as pd
pd.set_option('display.max_colwidth', None)
jeopardy = pd.read_csv('jeopardy.csv')
# RENAMING COLUMNS AFTER STRIPING UNNECESSARY WHITE SPACE
jeopardy.rename(columns={' Air Date': 'Air Date', ' Round': 'Round', ' Category': 'Category', ' Value': 'Value', ' Question': 'Question', ' Answer': 'Answer'}, inplace= True)
-
Write a function that filters the dataset for questions that contains all of the words in a list of words. For example, when the list
["King", "England"]was passed to our function, the function returned a DataFrame of 49 rows. Every row had the strings"King"and"England"somewhere in its" Question".Test your function by printing out the column containing the question of each row of the dataset.
# HELPER FUNCTION TO COMPARE THE ELEMENTS OF ONE LIST TO ANOTHER
def check_a_in_b(list_A, list_B):
for word_a in list_A:
result = False
for word_b in list_B:
if word_a.lower() == word_b.lower():
result = True
if result == False:
return result
return result
# USING THE HELPER FUNCTION check_a_in_b TO FILTER SPECIFIC QUESTIONS
def find_question(df, lst):
new = []
count = 0
for question in df:
lower_question = [word for word in question.split()]
target = check_a_in_b(lst, lower_question)
if target:
new.append(jeopardy.iloc[count])
count += 1
else:
count += 1
continue
result = pd.DataFrame(new)
return result
# LOWERING ALL STRING DATA TO THE SAME PLAYING FELID WITH LIST COMPREHENSION
# THEN USING ALL() FUNCTION TO FILTER SPECIFIC QUESTIONS
def find_question2(df, lst):
new = []
count = 0
for question in df:
lower_question = [word.lower() for word in question.split()]
lower_lst = [word.lower() for word in lst]
target = all(item in lower_question for item in lower_lst)
if target:
new.append(jeopardy.iloc[count])
count += 1
else:
count += 1
continue
result = pd.DataFrame(new)
return result
# LOWERING ALL STRING DATA TO THE SAME PLAYING FELID THEN USING LIST COMPREHENSION
# THEN USING ISSUBSET() FUNCTION TO FILTER SPECIFIC QUESTIONS
def find_question3(df, lst):
new = []
count = 0
for question in df:
lower_question = [word.lower() for word in question.split()]
lower_lst = [word.lower() for word in lst]
target = set(lower_lst).issubset(lower_question)
if target:
new.append(jeopardy.iloc[count])
count += 1
else:
count += 1
continue
result = pd.DataFrame(new)
return result
-
Test your original function with a few different sets of words to try to find some ways your function breaks. Edit your function so it is more robust.
For example, think about capitalization. We probably want to find questions that contain the word
"King"or"king".You may also want to check to make sure you don’t find rows that contain substrings of your given words. For example, our function found a question that didn’t contain the word
"king", however it did contain the word"viking"— it found the"king"inside"viking". Note that this also comes with some drawbacks — you would no longer find questions that contained words like"England's".
#TESTING OUT FUNCTION
king_england = find_question(jeopardy['Question'], ["King", "England"])
king_england2 = find_question2(jeopardy['Question'], ["King", "England"])
king_england3 = find_question3(jeopardy['Question'], ["King", "England's"])
print(king_england.Question.head(5)) # HELPER FUNCTION
print(king_england2.Question.head(5)) # ALL() FUNCTION
print(king_england3.Question.head(5)) # ISSUBSET() FUNCTION
-
We may want to eventually compute aggregate statistics, like
.mean()on the" Value"column. But right now, the values in that column are strings. Convert the" Value"column to floats. If you’d like to, you can create a new column with float values.Now that you can filter the dataset of question, use your new column that contains the float values of each question to find the “difficulty” of certain topics. For example, what is the average value of questions that contain the word
"King"?Make sure to use the dataset that contains the float values as the dataset you use in your filtering function.
#CONVERTING ROWS IN VALUE COLUMN TO TYPE FLOAT
jeopardy.Value = jeopardy.Value.apply(lambda row: float(str(row).replace('$', '').replace(',', '')))
print(jeopardy.Value.head())
#CREATING A NEW COLUMN WITH THE ELEMENTS OF VALUE COLUMN CONVERTED TO FLOATS
jeopardy['Float Value'] = jeopardy.Value.apply(lambda row: float(str(row).replace('$', '').replace(',', '')))
print(jeopardy['Float Value'].head())
#CREATING DATASET WITH ROWS THAT CONTAIN THE WORD KING IN THE QUESTION AND FINDING THE AVERAGE VALUE
just_king_mean = round(find_question(jeopardy.Question, ['king']).Value.mean(), 2)
print(just_king_mean)
- Write a function that returns the count of unique answers to all of the questions in a dataset. For example, after filtering the entire dataset to only questions containing the word
"King", we could then find all of the unique answers to those questions. The answer “Henry VIII” appeared 55 times and was the most common answer.
#CREATING FUNCTION THAT TAKES IN DATASET AND RETURNS UNIQUE VALUES IN SPECIFIC COLUMN
def unique_answer(data):
return data.value_counts()
king_england_unique = unique_answer(king_england.Answer)
print(king_england_unique)
- Explore from here! This is an incredibly rich dataset, and there are so many interesting things to discover. There are a few columns that we haven’t even started looking at yet. Here are some ideas on ways to continue working with this data:
- Investigate the ways in which questions change over time by filtering by the date. How many questions from the 90s use the word
"Computer"compared to questions from the 2000s? - Is there a connection between the round and the category? Are you more likely to find certain categories, like
"Literature"in Single Jeopardy or Double Jeopardy? - Build a system to quiz yourself. Grab random questions, and use the input function to get a response from the user. Check to see if that response was right or wrong.
def compare_data_by_date(data, year, lst):
two_thousands = data[data['Air Date'] >= str(year) + '-00-00']
nineties = data[data['Air Date'] < str(year) + '-00-00']
two_thousands_count = find_question(two_thousands.Question, lst)
nineties_count = find_question(nineties.Question, lst)
two_percent = round(len(two_thousands_count) / len(two_thousands) * 100, 2)
nine_percent = round(len(nineties_count) / len(nineties) * 100, 2)
print(str(two_percent) + '% of the questions from the ' + str(year) + 's contain the word ' + lst[0] + '.')
print(str(nine_percent) + '% of the questions from the ' + str(year) + 's contain the word ' + lst[0] + '.')
print('OR')
print(str(len(two_thousands_count)) + ' questions from the ' + str(year) + 's contain the word ' + lst[0] + '.')
print(str(len(nineties_count)) + ' questions from the ' + str(year) + 's contain the word ' + lst[0] + '.')
compare_data_by_date(jeopardy, 2000, ['Computer'])