import pandas as pd
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
try:
import seaborn
except ImportError:
pass
df = pd.read_csv("data/titanic.csv")
df.head()
Starting from reading this dataset, to answering questions about this data in a few lines of code:
What is the age distribution of the passengers?
df['Age'].hist()
How does the survival rate of the passengers differ between sexes?
df.groupby('Sex')[['Survived']].aggregate(lambda x: x.sum() / len(x))
Or how does it differ between the different classes?
df.groupby('Pclass')['Survived'].aggregate(lambda x: x.sum() / len(x)).plot(kind='bar')
Are young people more likely to survive?
df['Survived'].sum() / df['Survived'].count()
df25 = df[df['Age'] <= 25]
df25['Survived'].sum() / len(df25['Survived'])
All the needed functionality for the above examples will be explained throughout this tutorial.
Pandas provides two fundamental data objects, for 1D (Series
) and 2D data (DataFrame
).
A Series is a basic holder for one-dimensional labeled data. It can be created much as a NumPy array is created:
s = pd.Series([0.1, 0.2, 0.3, 0.4])
s
index
and values
¶The series has a built-in concept of an index, which by default is the numbers 0 through N - 1
s.index
You can access the underlying numpy array representation with the .values
attribute:
s.values
We can access series values via the index, just like for NumPy arrays:
s[0]
Unlike the NumPy array, though, this index can be something other than integers:
s2 = pd.Series(np.arange(4), index=['a', 'b', 'c', 'd'])
s2
s2['c']
In this way, a Series
object can be thought of as similar to an ordered dictionary mapping one typed value to another typed value.
In fact, it's possible to construct a series directly from a Python dictionary:
pop_dict = {'Germany': 81.3,
'Belgium': 11.3,
'France': 64.3,
'United Kingdom': 64.9,
'Netherlands': 16.9}
population = pd.Series(pop_dict)
population
We can index the populations like a dict as expected:
population['France']
but with the power of numpy arrays:
population * 1000
A DataFrame is a tablular data structure (multi-dimensional object to hold labeled data) comprised of rows and columns, akin to a spreadsheet, database table, or R's data.frame object. You can think of it as multiple Series object which share the same index.
One of the most common ways of creating a dataframe is from a dictionary of arrays or lists.
Note that in the IPython notebook, the dataframe will display in a rich HTML view:
data = {'country': ['Belgium', 'France', 'Germany', 'Netherlands', 'United Kingdom'],
'population': [11.3, 64.3, 81.3, 16.9, 64.9],
'area': [30510, 671308, 357050, 41526, 244820],
'capital': ['Brussels', 'Paris', 'Berlin', 'Amsterdam', 'London']}
countries = pd.DataFrame(data)
countries
A DataFrame has besides a index
attribute, also a columns
attribute:
countries.index
countries.columns
To check the data types of the different columns:
countries.dtypes
An overview of that information can be given with the info()
method:
countries.info()
Also a DataFrame has a values
attribute, but attention: when you have heterogeneous data, all values will be upcasted:
countries.values
If we don't like what the index looks like, we can reset it and set one of our columns:
countries = countries.set_index('country')
countries
To access a Series representing a column in the data, use typical indexing syntax:
countries['area']
As you play around with DataFrames, you'll notice that many operations which work on NumPy arrays will also work on dataframes.
# redefining the example objects
population = pd.Series({'Germany': 81.3, 'Belgium': 11.3, 'France': 64.3,
'United Kingdom': 64.9, 'Netherlands': 16.9})
countries = pd.DataFrame({'country': ['Belgium', 'France', 'Germany', 'Netherlands', 'United Kingdom'],
'population': [11.3, 64.3, 81.3, 16.9, 64.9],
'area': [30510, 671308, 357050, 41526, 244820],
'capital': ['Brussels', 'Paris', 'Berlin', 'Amsterdam', 'London']})
Just like with numpy arrays, many operations are element-wise:
population / 100
countries['population'] / countries['area']
Only, pay attention to alignment: operations between series will align on the index:
s1 = population[['Belgium', 'France']]
s2 = population[['France', 'Germany']]
s1
s2
s1 + s2
The average population number:
population.mean()
The minimum area:
countries['area'].min()
For dataframes, often only the numeric columns are included in the result:
countries.median()
Sorting the rows of the DataFrame according to the values in a column:
countries.sort_values('density', ascending=False)
One useful method to use is the describe
method, which computes summary statistics for each column:
countries.describe()
The plot
method can be used to quickly visualize the data in different ways:
countries.plot()
However, for this dataset, it does not say that much:
countries['population'].plot(kind='bar')
You can play with the kind
keyword: 'line', 'bar', 'hist', 'density', 'area', 'pie', 'scatter', 'hexbin'
A wide range of input/output formats are natively supported by pandas:
pd.read
states.to
.dropna()
, pd.isnull()
)concat
, join
)groupby
functionalitystack
, pivot
)There are many, many more interesting operations that can be done on Series and DataFrame objects, but rather than continue using this toy data, we'll instead move to a real-world example, and illustrate some of the advanced concepts along the way.
See the next notebooks!
© 2015, Stijn Van Hoey and Joris Van den Bossche (mailto:stijnvanhoey@gmail.com, mailto:jorisvandenbossche@gmail.com). Licensed under CC BY 4.0 Creative Commons
This notebook is partly based on material of Jake Vanderplas (https://github.com/jakevdp/OsloWorkshop2014).