So a bit of a general question / ask for advice about using python for data analysis.
So say I have a script like this:
df = pd.read_sql_query('SELECT * from DB', connection)
I run this, but then think… ah I want to view some certain data so I add some code and run script again
foo = df["bar"]
Then I think, I want to actually print this variable, so I add more code and run it again
So at this point I have run 3 seperate requests to my DB to retrieve the exact same data. I could see this manner of coding quickly becoming an issue if im importing a huge dataset or if im using an API with a request limit.
How do people generally get around this? My only thoughts were to use an IDE like spyder and make manipulations to my imported variable using the iPython console. This seems like a bit of a work around though.
What is “best practice”? Is there a way to keep a variable in python memory while you are amending and rerunning the program to manipulate this data?