Explainable Outlier Detection in Titanic dataset¶
This short notebook illustrates basic usage of the OutlierTree library for explainable outlier detection using the Titanic dataset. For more details, you can check the package's documentation here.
The dataset is very popular and can be downloaded from different sources, such as Kaggle or many university webpages. This notebook took it from the following link: http://biostat.mc.vanderbilt.edu/wiki/pub/Main/DataSets/titanic3.csv
Loading the raw data¶
In [1]:
import numpy as np, pandas as pd
from outliertree import OutlierTree
## Read the raw data, downloaded from here:
## https://github.com/jbryer/CompStats/raw/master/Data/titanic3.csv
titanic = pd.read_csv("titanic3.csv")
titanic.head()
Out[1]:
Pre-processing the data¶
In [2]:
## Capitalize column names and some values for easier reading
titanic.columns = titanic.columns.str.capitalize()
titanic = titanic.rename(columns = {"Sibsp" : "SibSp"})
titanic["Sex"] = titanic["Sex"].str.capitalize()
## Convert 'survived' to yes/no for easier reading
titanic["Survived"] = titanic["Survived"].astype("category").replace({1:"Yes", 0:"No"})
## Some columns are not useful, such as name (an ID), ticket number (another ID),
## or destination (too many values, many non-repeated)
cols_drop = ["Name", "Ticket", "Home.dest"]
titanic = titanic.drop(cols_drop, axis=1)
## Ordinal columns need to be passed as ordered categoricals
cols_ord = ["Pclass", "Parch", "SibSp"]
for col in cols_ord:
titanic[col] = pd.Categorical(titanic[col], ordered=True)
titanic.head()
Out[2]:
Fitting a model¶
In [3]:
## Fit model with default hyperparameters
otree = OutlierTree()
otree.fit(titanic)
Out[3]:
Examining the results more closely¶
In [4]:
## Double-check the data (last 2 outliers)
titanic.loc[[1146, 1163]]
Out[4]:
In [5]:
## Distribution of the group from which those two outliers were flagged
%matplotlib inline
import matplotlib.pyplot as plt
titanic.loc[
(titanic.Pclass == 3) &
(titanic.SibSp == 0) &
(titanic.Embarked == "Q")
] .Fare.hist(bins=50, color="navy", edgecolor='black', linewidth=1.2)
plt.xlabel("Fare", fontsize=15)
plt.ylabel("Frequency", fontsize=15)
plt.title("Distribution of Fare within cluster", fontsize=20)
plt.show()
In [6]:
## Get the outliers in a manipulable format
otree.predict(titanic).loc[[1146, 1163]]
Out[6]:
In [7]:
## To programatically get all the outliers that were flagged
pred = otree.predict(titanic)
pred.loc[~pred.outlier_score.isnull()]
Out[7]:
In [8]:
## To print selected rows only
otree.print_outliers(pred.loc[[1146]])
Trying different hyperparameters¶
In [9]:
## In order to flag more outliers, one can also experiment
## with lowering the threshold hyperparameters
OutlierTree(z_outlier=6.).fit(titanic, outliers_print=5)
Out[9]:
In [10]:
## One can also lower the gain threshold, but this tends
## to result in more spurious outliers which come from
## not-so-good splits (not recommended)
OutlierTree(z_outlier=6, min_gain=1e-6).fit(titanic, outliers_print=5)
Out[10]: