Section 1.2 — Data in practice

This notebook contains all the code from Section 1.2 Data in practice of the No Bullshit Guide to Statistics.

Notebook setup¶

Getting started with JupyterLab¶

Download and install JupyterLab Desktop¶

Follow instructions in the Python tutorial to install JupyterLab Desktop on your computer.

Download the noBSstats notebooks and datasets¶

Go to URL https://github.com/minireference/noBSstats and use the Code dropdown button to download the ZIP archive of the entire repository.

After downloading the ZIP archive, double-click on the file to extract its contents, and move the resulting folder noBSstats to a location on your computer where you normally keep your documents.

The ZIP archive includes all the datasets and computational notebooks for the book. Use the File browser pane on the right to navigate to the location where you saved the noBSstats folder and explore the subfolders datasets, notebooks, exercises, tutorials, etc.

Datasets for the book¶

The datasets folder contains all the datasets used in examples and exercises throughout the book.

Interactive notebooks for each section¶

The notebooks folder contains the jupyter notebooks associated with each section of the book, similar to the one you’re currently looking at.

Exercises notebooks¶

The exercises folder contains starter notebooks for the exercises in each section.

TODO: insert here updated list of exercise notebooks

Data management with pandas¶

The first step is to import the pandas library. We’ll follow the standard convention of importing the pandas module under the alias pd.

Loading datasets¶

Players dataset¶

Consider the data file players.csv located in the datasets directory. The file extension .csv tells us the file contains text data formatted as Comma-Separated Values (CSV). We can use the command %pycat to print the raw contents of the this file.

We see the file contains 13 lines of text, and each line contains—as promised by the .csv file extension—values separated by commas. The first line in the data file is called the “header” and contains the names of the variable names.

We can create a the data frame object from the players dataset located at datasets/players.csv by calling the function pd.read_csv.

Data frame properties¶

What type of object is players ?

The players data frame object has a bunch of useful properties (attributes) and functions (methods) “attached” to it, which we can access using the dot syntax.

The shape of the players data frame¶

The rows index¶

The columns index¶

Exploring data frame objects¶

Data types¶

Accessing values in a DataFrame¶

Selecting individual values¶

The player with username emil has index 2 in the data frame. To see the value of the points variable for the player emil, we use the following expression based on the .loc[] selector syntax.

Selecting entire rows¶

Let’s now select all the measurements we have for Sanjay (the player at index 6).

Selecting entire columns¶

We use the square brackets to select columns from a data frame. For example, this is how we extract the "age" column from the players data frame.

Selecting multiple columns¶

We can select multiple columns by using list of column names inside the square brackets.

Statistical calculations using pandas¶

Let’s extract the values from the "age" column from the players data frame and store them as new variable called ages. We intentionally choose the name ages (plural of the column name) to remember where the data comes from.

The variable ages is a pandas series object.

We can access individual values within the series ages using the square brackets.

Series attributes¶

The pandas series ages has the same index as the players data frame.

The series ages also “remembers” the name of the column from which it was extracted.

We sometimes want to see the data without the index. We can do this by accessing the .values attribute of the series.

Series methods¶

Alternatively, since series objects are list-like, we can use the Python build in function len to find the length of the series.

The average value of a list of $n$ values $\mathbf{x} = [x_1, x_2, \ldots, x_n]$ is computed using the formula $\overline{\mathbf{x}} = \tfrac{1}{n}\!\left( x_1 + x_2 + \cdots + x_n \right)$. This formula says that the average is computed by summing together all the values in the list $\mathbf{x}$ and dividing by the length of the list $n$.

The expression for computing the average age using pandas methods is as follows:

An equivalent, more direct, way to compute the arithmetic mean of the values in the series ages is to call its .mean() method.

The standard deviation (dispersion from the mean) is another common statistic that we might want to calculate for a variable in a dataset.

Pandas series and data frames objects have numerous other methods for computing numerical data summaries, which are called the descriptive statistics of the variable. We’ll learn more about those in Section 1.3 Descriptive statistics.

Selecting only certain rows (filtering)¶

We often want to select a subset of the rows of a data frame that fit one or more criteria. This is equivalent to “filtering out” the rows that don’t satisfy these criteria. We use a two-step procedure for this:

• Step 1: Build a “selection mask” series that consists of boolean values (True or False).

• Step 2: Select the subset of rows from the data frame using the mask. The result is a new data frame that contains only the rows that correspond to the True values in the selection mask.

To select only rows where ezlvl is 1, we first build the boolean selection mask (Step 1)...

... then select the rows using the mask (Step 2).

The above two step process can be combined into a more compact expression:

Sorting data frames¶

Ranking (optional material)¶

We can also rank the players according to the time variable by using the method .rank() on the "time" column.

The rank of an element in a list tells us the position it appears in when the list is sorted. We see from the players-ranked-by-time results that the player at index 7 has rank 1 (first), and the player at index 5 is ranked 12 (last).

Data visualization with seaborn¶

The first step is to import the seaboarn module under the alias sns.

All the seaborn functions are now available under sns..

Strip plot of the time variable¶

To generate a strip plot, we pass the data frame players as the data argument to the seaborn function sns.stripplot, and specify the column name "time" (in quotes) as the x argument.

We can enhance the strip plot by mapping the ezlvl variable to the colour (hue) of the points in the plot.

Studying the effect of ezlvl on time¶

The players dataset was collected as part of an experiment designed to answer the question “Does the easy first level lead to an improvement in user retention?” We want to compare the time variable (total time players spent in the game) of players who were shown the “easy level” version of the game (ezlvl==1) to the control group of played who played the regular vesion of the game (ezlvl==0).

The mean time in the intervention group is:

The mean time in the control group is:

Let’s generate a strip plot of the time variable for the two groups of players.

Studying the relationship between age and time¶

The secondary research question, is to look for a correlation between the age variable and the time variable.

We can also create a linear regression plot using the regplot function.

We’ll learn more about linear regression in Chapter 4.

Real-world datasets¶

Imagine you’re a data scientist consulting with various clients. Clients come to you with datasets and real-world questions they want to answer using statistical analysis. The following table shows the complete list of the datasets that we’ll use in examples and explanations in the rest of the book. The last column of the table tells us the sections of the book where each dataset will be discussed.

Let’s briefly describe the background story behind each dataset and state the statistical question that each client is interested in answering.

Apple weights¶

Alice runs an apple orchard. She collected a sample from the apples harvested this year (the population) and sent you the data in a CSV file called apples.csv.

You start by loading the data into pandas and looking at its characteristics.

The first three observations in the apples dataset are:

The sample mean is:

Electricity prices¶

Bob recently bought an electric car. He doesn’t have a charging station for his car at home, so he goes to public charging stations to recharge the car’s batteries. Bob lives downtown, so he can go either to the East End or West End of the city for charging. He wants to know which side of the city has cheaper prices. Are electricity prices cheaper in the East End or the West End of the city?

To study this question, Bob collected electricity prices of East End and West End charging stations from a local price comparison website and provided you the prices in the dataset epriceswide.csv.

You note that the data frame epriceswide contains two observations per row: one price from an East End charging station and another one from a West End charging station. You decide to reshape the data frame so that each observation appears on a separate row.

You can use the pandas method .melt() to convert the epriceswide data frame from “wide” format into “long” format, with one observation per row. The method .melt() uses the option var_name to specify the name of the variable that is encoded in the column positions: the location (East or West), and the optionvalue_name to specify the name of the variable stored in the individual cells, which is the price.

The .melt operation transformed the implicit “which column is the data in” information into an explicit loc variable stored in a separate column. Each row in the transformed data frame contains only a single observation.

You decide to save the transformed data frame eprices as a new data file datasets/eprices.csv, which you plan to use for all the subsequent analyses of Bob’s dataset.

You then reload data from the file datasets/eprices.csv to get back the same eprices data frame.

The first three observations in the electricity prices dataset are:

Let’s create a strip plot of electricity prices along the $x$-axis and use the loc variable to control the $y$-position and colour of the points.

The average price in the West End is:

The average price in the East End is:

Students’ scores¶

Charlotte is a science teacher who wants to test the effectiveness of a new teaching method in which material is presented in the form of a “scientific debate”. Student actors initially express “wrong” opinions, which are then corrected by presenting the “correct” way to think about science concepts. This type of teaching is in contrast to the usual lecture method, in which the teacher presents only the correct facts.

To compare the effectiveness of the two teaching methods, she has prepared two variants of her course:

• In the lecture variant, the video lessons present the material in the usual lecture format that includes only correct facts and explanations.

• In the debate variant, the same material is covered through video lessons in which student actors express multiple points of view, including common misconceptions.

Except for the different video lessons, the two variants of the course are identical: they cover the same topics, use the same total lecture time, and test students’ knowledge using the same assessment items.

Let’s load the data file students.csv into pandas and see what it looks like.

The first five observations in the students dataset are:

We’re interested in comparing the two variants of the course, so we can generate a strip plot of the score variable, for the two groups defined by the curriculum variable.

Kombucha volumes¶

Khalid is responsible for the production line at a kombucha brewing company. He needs to make sure the volume of kombucha that goes into each bottle is exactly 1 litre (1000 ml), but because of day-to-day variations in the fermentation process, production batches may end up with under-filled or over-filled bottles. Sending such irregular batches to clients will cause problems for the company, so Khalid wants to find a way to detect when the brewing and bottling process is not working as expected.

Khalid compiled the dataset kombucha.csv, which contains the volume measurements from samples taken from 10 different production batches,

The first three observations in the kombucha dataset are:

Let’s generate a combined strip plot of the observations from the different batches so that we can visually inspect the data.

Average volume of the sample from Batch 01¶

Doctors’ sleep study¶

Dan is a data analyst working at the Ministry of Health. His current assignment is to look for ways to improve the health of family doctors. He collected the doctors dataset (doctors.csv), which contains data about the demographics, life habits, and health metrics of 224 family doctors that Dan randomly selected from the populations of family doctors in the country.

The first three observations in the doctors dataset are:

Dan is interested in comparing the sleep scores of doctors in rural and urban locations, so he starts by generating a strip plot of the score variable for the two values of the location variable.

Average sleep scores for doctors in different locations¶

We see doctors in rural locations get better sleep, on average, than doctors in urban locations.

Website visitors¶

Vanessa runs an e-commerce website and is about to launch a new design for the homepage. She wants to know if the new design is better or worse than the current design. Vanessa has access to the server logs from her website and is able to collect data about which visitors clicked the BUY NOW button and bought something. The term conversion is used when a visitor buys something, meaning they are “converted” from visitor to client. The conversion rate is the proportion of website visitors that become clients.

Vanessa performed an experiment to check if the new website design is better than the current design when it comes to getting visitors to click the BUY NOW button. For the 2000 new visitors that the site received during the previous month, Vanessa randomly sent half of them to the new design (A for alternative), and the other half to the old design (B for baseline). She also recorded if a visitor bought a product during their visit to the website.

The data consists of 2000 observations from visitors to the website from unique IP addresses. For each visitor, the column version contains which design they were presented with, and the column bought records whether the visitor purchased something or not. You use the usual pandas commands to load the dataset visitors.csv to inspect it.

The first five observations in the visitors dataset are:

The average (mean) conversion rate for version A of the website is:

The average (mean) conversion rate for version B of the website is:

We can represent the two conversion rates using a bar plot.

The black lines represent the estimates of the uncertainty of the true conversion rates that seaborn automatically computed for us. Specifically, the black lines represent $95\%$ confidence intervals for the true proportions. We’ll learn about confidence intervals later in the book (Section 3.2).

Explanations¶

Using a variable as the index¶

I want to show you an example of a data frame that uses non-numeric index. For example, we can use the players’ username as the index.

The index of the data frame players_by_name consists of strings. Here are the row-labels we must use to access the data observations.

We can now access players by username.

This examples is to highlight the fact that pandas data frames are more powerful than spreadsheets, since we can use arbitrary labels for the rows.

Discussion¶

Data extraction¶

See Appendix D: Pandas tutorial.

Data transformations¶

See Appendix D: Pandas tutorial.

Tidy data¶

The concept of tidy data is a convention for organizing datasets that makes statistical calculations and visualizations easy to perform. Just like a tidy workplace, a tidy dataset makes it easy to find what you’re looking for. A data frame is organized according to the tidy data format if it has the following characteristics:

• Each column contains the values for one variable.

• Each row contains the values for one observation.

• Each data cell contains a single value.

This specific organization of data into rows and columns makes it easy to perform statistical calculations on arbitrary subsets of the data, and allows us to create seaborn plots by mapping column names to visual aspects of the plot, as we saw many times in this section.

The students dataset is tidy¶

The initial electricity prices dataset was not tidy¶

The dataset epriceswide.csv that Bob sent you was not tidy, because each row contains multiple observations: an price from the East and and unrelated price from the West.

We called pandas the method .melt(var_name="loc", value_name="price") on the data frame epriceswide to convert it to the tidy data frame eprices, which we saved in the data file datasets/eprices.csv.

Data cleaning¶

See Appendix D: Pandas tutorial.

Links¶

TODO: import from Appendix D