# Lesson 3

Get Data - Our data set will consist of an Excel file containing customer counts per date. We will learn how to read in the excel file for processing.
Prepare Data - The data is an irregular time series having duplicate dates. We will be challenged in compressing the data and comming up with next years forecasted customer count.
Analyze Data - We use graphs to visualize trends and spot outliers. Some built in computatiopnal tools will be used to calculate next years forecasted customer count.
Present Data - The results will be graphed.

NOTE: Make sure you have looked through all previous lessons as the knowledge learned in previous lessons will be needed for this exercise.

In [1]:
# Import libraries
from pandas import ExcelFile, DataFrame, concat, date_range
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np

In [2]:
print 'Pandas version: ' + pd.__version__

Pandas version: 0.11.0



We will be creating our own test data for analysis.

In [3]:
# Function to generate test data
def CreateDataSet(Number=1):

Output = []

for i in range(Number):

# Create a weekly (mondays) date range
rng = date_range(start='1/1/2009', end='12/31/2012', freq='W@MON')

# Create random data
data = np.random.randint(low=25,high=1000,size=len(rng))

# Status pool
status = [1,2,3]

# Make a random list of statuses
seed(i)
random_status = [status[randint(low=0,high=len(status))] for i in range(len(rng))]

# State pool
states = ['GA','FL','fl','NY','NJ','TX']

# Make a random list of states
random_states = [states[randint(low=0,high=len(states))] for i in range(len(rng))]

Output.extend(zip(random_states, random_status, data, rng))

return Output


Now that we have a function to generate our test data, lets create some data and stick it into a dataframe.

In [4]:
dataset = CreateDataSet(4)
df = DataFrame(data=dataset, columns=['State','Status','CustomerCount','StatusDate'])
df

Out[4]:
<class 'pandas.core.frame.DataFrame'>
Int64Index: 836 entries, 0 to 835
Data columns (total 4 columns):
State            836  non-null values
Status           836  non-null values
CustomerCount    836  non-null values
StatusDate       836  non-null values
dtypes: datetime64[ns](1), int64(2), object(1)


We are now going to save this dataframe into an Excel file, to then bring it back to a dataframe. We simply do this to show you how to read and write to excel files.

We do not write the index values of the dataframe to the excel file since they are no meant to be part of our initial test data set.

In [5]:
# Save results to excel
df.to_excel('Lesson3.xlsx', index=False)
print 'Done'

Done



# Grab Data from Excel

We will be using the ExcelFile function and the parse function to read in data from an excel file. The ExcelFile function creates an object and parse will help with the actually parsing of the file. Lets take a loook at both of these.

ExcelFile:
Parameters
path : string or file-like objec, Path to xls file
kind : {'xls', 'xlsx', None}, default None
Definition:ExcelFile(self, path_or_buf)

parse:
Definition: ExcelFile.parse(self, sheetname, header=0, skiprows=None, skip_footer=0, index_col=None, parse_cols=None, parse_dates=False, date_parser=None, na_values=None, thousands=None, chunksize=None, **kwds)
Docstring: Read Excel table into DataFrame

In [6]:
ExcelFile?

In [7]:
ExcelFile.parse?


Note: The location on the excel file will be in the same folder as the notebook unless specified otherwise.

In [8]:
# Location of file
Location = r'C:\Users\hdrojas\.xy\startups\Lesson3.xlsx'

# Create ExcelFile object
xlsx = ExcelFile(Location)

# Parse a specific sheet
df = xlsx.parse('sheet1',index_col='StatusDate')
df.dtypes

Out[8]:
State             object
Status           float64
CustomerCount    float64
dtype: object

In [9]:
df.index

Out[9]:
<class 'pandas.tseries.index.DatetimeIndex'>
[2009-01-05 00:00:00, ..., 2012-12-31 00:00:00]
Length: 836, Freq: None, Timezone: None


As you can see the first problem we have here is that the column Status and the column CustomerCount are of data type float64. We would rather have these columns be an int data type. Below we will simply solve this issue by casting the columns to their appropriate type.

In [10]:
# Convert data types
df.Status = df.Status.astype('int')
df.CustomerCount = df.CustomerCount.astype('int')
print 'Data Types'
print df.dtypes

Data Types
State            object
Status            int32
CustomerCount     int32
dtype: object


In [11]:
df.head()

Out[11]:
State Status CustomerCount
StatusDate
2009-01-05 NY 1 721
2009-01-12 GA 2 86
2009-01-19 NY 1 441
2009-01-26 GA 2 992
2009-02-02 NJ 2 614

# Prepare Data

This section attempts to clean up the data for analysis.
1. Make sure the state column is all in upper case
2. Only select records where the account status is equal to "1"
3. Merge (NJ and NY) to NY in the state column
4. Remove any outliers (any odd results in the data set)

Lets take a quick look on how some of the State values are upper case and some are lower case

In [12]:
df['State'].unique()

Out[12]:
array([NY, GA, NJ, fl, TX, FL], dtype=object)


To convert all the State values to upper case we will use the upper() function and the dataframe's apply attribute. The lambda function simply will apply the upper function to each value in the State column.

In [13]:
# Clean State Column, convert to upper case
df.State = df.State.apply(lambda x: x.upper())

In [14]:
df['State'].unique()

Out[14]:
array([NY, GA, NJ, FL, TX], dtype=object)

In [15]:
# Only grab where Status == 1
df = df[df['Status'] == 1]


To turn the NJ states to NY we simply.

[df.State == 'NJ'] - Find all records in the State column where they are equal to NJ.
df.State[df.State == 'NJ'] = 'NY' - For all records in the State column where they are equal to NJ, replace them with NY.

In [16]:
# Convert NJ to NY
df.State[df.State == 'NJ'] = 'NY'


Now we can see we have a much cleaner data set to work with.

In [17]:
df['State'].unique()

Out[17]:
array([NY, TX, GA, FL], dtype=object)


At this point we may want to graph the data to check for any outliers or inconsistencies in the data. We will be using the plot() attribute of the dataframe.

As you can see from the graph below it is not very conclusive and is probably a sign that we need to perform some more data preparation.

In [18]:
df['CustomerCount'].plot()

Out[18]:
<matplotlib.axes.AxesSubplot at 0x6cf0b90>


If we take a look at the data, we begin to realize that there are multiple values per State, StatusDate, and Status. It is possible that this means the data you are working with is dirty/bad/inaccurate, but we will assume otherwise. We can assume this data set is a subset of a bigger data set and if we simply add the values in the CustomerCount column per State, StatusDate, and Status we will get the Total Customer Count per day.

In [19]:
sortdf = df[df['State']=='NY'].sort(axis=0)

Out[19]:
State Status CustomerCount
StatusDate
2009-01-05 NY 1 721
2009-01-12 NY 1 368
2009-01-12 NY 1 103
2009-01-19 NY 1 441
2009-01-26 NY 1 408

Our task is now to create a new dataframe that compresses the data so we have daily customer counts per State and StatusDate. We can ignore the Status column since all the values in this column are of value 1. To accomplish this we will use the dataframe's functions groupby and sum().

Note that we had to use reset_index . If we did not, we would not have been able to group by both the State and the StatusDate since the groupby function expects only columns as inputs. The reset_index function will bring the index StatusDate back to a column in the dataframe.

In [20]:
# Group by State and StatusDate
Daily = df.reset_index().groupby(['State','StatusDate']).sum()

Out[20]:
Status CustomerCount
State StatusDate
FL 2009-02-02 1 385
2009-02-09 1 125
2009-02-16 1 378
2009-03-02 1 722
2009-05-18 1 962

The State and StatusDate columns are automatically placed in the index of the Daily dataframe. You can think of the index as the primary key of a database table but without the constraint of having unique values. Columns in the index as you will see allow us to easily select, plot, and perform calculations on the data.

Below we delete the Status column since it is all equal to one and no longer necessary.

In [21]:
del Daily['Status']

Out[21]:
CustomerCount
State StatusDate
FL 2009-02-02 385
2009-02-09 125
2009-02-16 378
2009-03-02 722
2009-05-18 962
In [22]:
# What is the index of the dataframe
Daily.index

Out[22]:
MultiIndex
[FL  2009-02-02,     2009-02-09,     2009-02-16,     2009-03-02,     2009-05-18,     2009-06-08,     2009-06-15,     2009-06-22,     2009-07-06,     2009-07-27,     2009-08-03,     2009-09-14,     2009-09-28,     2009-10-19,     2009-11-23,     2009-11-30,     2010-01-04,     2010-01-11,     2010-02-01,     2010-02-15,     2010-03-15,     2010-03-22,     2010-04-12,     2010-04-19,     2010-04-26,     2010-05-03,     2010-05-10,     2010-05-17,     2010-06-07,     2010-06-21,     2010-06-28,     2010-07-05,     2010-08-30,     2010-10-11,     2010-10-18,     2010-10-25,     2010-11-01,     2010-11-15,     2010-11-29,     2010-12-27,     2011-01-03,     2011-01-10,     2011-01-24,     2011-02-07,     2011-03-07,     2011-03-14,     2011-03-28,     2011-04-04,     2011-04-18,     2011-04-25, ..., NY  2012-10-22,     2012-11-12,     2012-11-26,     2012-12-03, TX  2009-01-05,     2009-01-19,     2009-03-02,     2009-03-16,     2009-04-13,     2009-04-20,     2009-06-01,     2009-08-03,     2009-08-31,     2009-09-21,     2009-12-14,     2010-01-04,     2010-02-15,     2010-04-19,     2010-05-31,     2010-06-07,     2010-06-14,     2010-06-28,     2010-07-05,     2010-08-09,     2010-08-23,     2010-09-06,     2010-10-04,     2010-11-01,     2010-11-08,     2010-12-13,     2011-01-17,     2011-02-14,     2011-02-28,     2011-03-14,     2011-05-16,     2011-06-13,     2011-09-12,     2011-09-26,     2011-10-17,     2011-11-07,     2011-11-21,     2011-12-05,     2012-01-30,     2012-03-05,     2012-03-26,     2012-06-04,     2012-07-02,     2012-07-30,     2012-10-08,     2012-11-12]

In [23]:
# Select the State index
Daily.index.levels[0]

Out[23]:
Index([FL, GA, NY, TX], dtype=object)

In [24]:
# Select the StatusDate index
Daily.index.levels[1]

Out[24]:
<class 'pandas.tseries.index.DatetimeIndex'>
[2009-01-05 00:00:00, ..., 2012-12-31 00:00:00]
Length: 181, Freq: None, Timezone: None


Lets now plot the data per State.

As you can see by breaking the graph up by the State column we have a much clearer picture on how the data looks like. Can you spot any outliers?

In [25]:
Daily.loc['FL'].plot()
Daily.loc['GA'].plot()
Daily.loc['NY'].plot()
Daily.loc['TX'].plot()

Out[25]:
<matplotlib.axes.AxesSubplot at 0x5850bd0>


We can also just plot the data on a specific date, like 2012. We can now clearly see that the data for these states is all over the place. since the data consist of weekly customer counts, the variability of the data seems suspect. For this tutorial we will assume bad data and proceed.

In [26]:
Daily.loc['FL']['2012':].plot()
Daily.loc['GA']['2012':].plot()
Daily.loc['NY']['2012':].plot()
Daily.loc['TX']['2012':].plot()

Out[26]:
<matplotlib.axes.AxesSubplot at 0x6d49530>


We will assume that per month the customer count should remain relatively steady. Any data outside a specific range in that month will be removed from the data set. The final result should have smooth graphs with no spikes.

StateYearMonth - Here we group by State, Year of StatusDate, and Month of StatusDate.
Daily['Outlier'] - A boolean (True or False) value letting us know if the value in the CustomerCount column is ouside the acceptable range.

We will be using the attribute transform instead of apply. The reason is that transform will keep the shape(# of rows and columns) of the dataframe the same and apply will not. By looking at the previous graphs, we can realize they are not resembling a gaussian distribution, this means we cannot use summary statistics like the mean and stDev. We use percentiles instead. Note that we run the risk of eliminating good data.

In [27]:
# Calculate Outliers
StateYearMonth = Daily.groupby([Daily.index.get_level_values(0), Daily.index.get_level_values(1).year, Daily.index.get_level_values(1).month])
Daily['Lower'] = StateYearMonth['CustomerCount'].transform( lambda x: x.quantile(q=.25) - (1.5*x.quantile(q=.75)-x.quantile(q=.25)) )
Daily['Upper'] = StateYearMonth['CustomerCount'].transform( lambda x: x.quantile(q=.75) + (1.5*x.quantile(q=.75)-x.quantile(q=.25)) )
Daily['Outlier'] = (Daily['CustomerCount'] < Daily['Lower']) | (Daily['CustomerCount'] > Daily['Upper'])

# Remove Ouliers
Daily = Daily[Daily['Outlier'] == False]


The dataframe named Daily will hold customer counts that have been aggregated per day. The original data (df) has multiple records per day. We are left with a data set that is indexed by both the state and the StatusDate. The Oulier column should be equal to false signifying that the record is not an outlier.

In [28]:
Daily.head()

Out[28]:
CustomerCount Lower Upper Outlier
State StatusDate
FL 2009-02-02 385 -69 702 False
2009-02-09 125 -69 702 False
2009-02-16 378 -69 702 False
2009-03-02 722 361 1083 False
2009-05-18 962 481 1443 False

We create a seperate dataframe named ALL which groups the Daily dataframe by StatusDate. We are essentially getting rid of the State column. The Max column represents the maximum customer count per month. The Max column is used to smooth out the graph.

In [29]:
# Combine all markets

# Get the max customer count by Date
ALL = DataFrame(Daily['CustomerCount'].groupby(Daily.index.get_level_values(1)).sum())
ALL.columns = ['CustomerCount'] # rename column

# Group by Year and Month
YearMonth = ALL.groupby([lambda x: x.year, lambda x: x.month])

# What is the max customer count per Year and Month
ALL['Max'] = YearMonth['CustomerCount'].transform(lambda x: x.max())

Out[29]:
CustomerCount Max
StatusDate
2009-01-05 1525 1551
2009-01-12 471 1551
2009-01-19 1551 1551
2009-01-26 408 1551
2009-02-02 385 977

As you can see from the ALL dataframe above, in the month of January 2009, the maximum customer count was 1,743. If we used apply, we would have a dataframe with (Year and Month) as the index and just the Max column with the value of 1,743.

There is also an interest to gauge if the current customer counts were reaching certain goals the company had established. The task here is to visually show if the current customer counts are meeting the goals listed below. We will call the goals BHAG (Big Hairy Annual Goal).

• 12/31/2011 - 1,000 customers
• 12/31/2012 - 2,000 customers
• 12/31/2013 - 3,000 customers

We will be using the date_range function to create our dates.

Definition: date_range(start=None, end=None, periods=None, freq='D', tz=None, normalize=False, name=None)
Docstring: Return a fixed frequency datetime index, with day (calendar) as the default frequency

By choosing the frequency to be A or annual we will be able to get the three target dates from above.

In [30]:
date_range?

In [31]:
# Create the BHAG dataframe
data = [1000,2000,3000]
idx = date_range(start='12/31/2011', end='12/31/2013', freq='A')
BHAG = DataFrame(data, index=idx, columns=['BHAG'])
BHAG

Out[31]:
BHAG
2011-12-31 1000
2012-12-31 2000
2013-12-31 3000

Combining dataframes as we have learned in previous lesson is made simple using the concat function. Remember when we choose axis = 0 we are appending row wise.

In [32]:
# Combine the BHAG and the ALL data set
combined = concat([ALL,BHAG], axis=0)
combined = combined.sort(axis=0)
combined.tail()

Out[32]:
BHAG CustomerCount Max
2012-12-10 NaN 73 633
2012-12-17 NaN 138 633
2012-12-31 2000 NaN NaN
2012-12-31 NaN 633 633
2013-12-31 3000 NaN NaN
In [33]:
fig, axes = plt.subplots(figsize=(10, 5))
combined['Max'].plot(color='blue', label='All Markets')
plt.legend(loc='best')

Out[33]:
<matplotlib.legend.Legend at 0x593ee90>


There was also a need to forecast next year's customer count and we can do this in a couple of simple steps. We will first group the combined dataframe by Year and place the maximum customer count for that year. This will give us one row per Year.

In [34]:
# Group by Year and then get the max value per year
Year = combined.groupby(lambda x: x.year).max()
Year

Out[34]:
BHAG CustomerCount Max
2009 NaN 2421 2421
2010 NaN 2598 2598
2011 1000 2604 2604
2012 2000 2420 2420
2013 3000 NaN NaN
In [35]:
# Add a column representing the percent change per year
Year['YR_PCT_Change'] = Year['Max'].pct_change(periods=1)
Year

Out[35]:
BHAG CustomerCount Max YR_PCT_Change
2009 NaN 2421 2421 NaN
2010 NaN 2598 2598 0.073110
2011 1000 2604 2604 0.002309
2012 2000 2420 2420 -0.070661
2013 3000 NaN NaN NaN

To get next year's end customer count we will assume our current growth rate remains constant. We then will increase this years customer count by that amount and that will be our forecast for next year.

In [36]:
(1 + Year.ix[2012,'YR_PCT_Change']) * Year.ix[2012,'Max']

Out[36]:
2249.0015360983102


# Present Data

Create individual Graphs per Market.

In [37]:
# First Graph
ALL['Max'].plot(figsize=(10, 5));plt.title('ALL Markets')

# Last four Graphs
fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(20, 10))
fig.subplots_adjust(hspace=1.0) ## Create space between plots


<matplotlib.text.Text at 0x73e1c10>