Previously we worked on visualizing the iris data by plotting pairs of dimensions by trial and error, until we arrived at the best pair of dimensions for our dataset. Here we will use an unsupervised dimensionality reduction algorithm to accomplish this more automatically.
By the end of this section you will
Dimensionality reduction is the task of deriving a set of new artificial features that is smaller than the original feature set while retaining most of the variance of the original data. Here we'll use a common but powerful dimensionality reduction technique called Principal Component Analysis (PCA). We'll perform PCA on the iris dataset that we saw before:
from sklearn.datasets import load_iris
iris = load_iris()
X = iris.data
y = iris.target
PCA is performed using linear combinations of the original features using a truncated Singular Value Decomposition of the matrix X so as to project the data onto a base of the top singular vectors. If the number of retained components is 2 or 3, PCA can be used to visualize the dataset.
from sklearn.decomposition import PCA
pca = PCA(n_components=2, whiten=True)
pca.fit(X)
Once fitted, the pca model exposes the singular vectors in the components_ attribute:
pca.components_
Other attributes are available as well:
pca.explained_variance_ratio_
pca.explained_variance_ratio_.sum()
Let us project the iris dataset along those first two dimensions:
X_pca = pca.transform(X)
PCA normalizes
and whitens
the data, which means that the data
is now centered on both components with unit variance:
import numpy as np
np.round(X_pca.mean(axis=0), decimals=5)
np.round(X_pca.std(axis=0), decimals=5)
Furthermore, the samples components do no longer carry any linear correlation:
np.corrcoef(X_pca.T)
We can visualize the projection using pylab, but first let's make sure our ipython notebook is in pylab inline mode
%pylab inline
Now we can visualize the results using the following utility function:
import pylab as pl
from itertools import cycle
def plot_PCA_2D(data, target, target_names):
colors = cycle('rgbcmykw')
target_ids = range(len(target_names))
pl.figure()
for i, c, label in zip(target_ids, colors, target_names):
pl.scatter(data[target == i, 0], data[target == i, 1],
c=c, label=label)
pl.legend()
Now calling this function for our data, we see the plot:
plot_PCA_2D(X_pca, iris.target, iris.target_names)
Note that this projection was determined without any information about the labels (represented by the colors): this is the sense in which the learning is unsupervised. Nevertheless, we see that the projection gives us insight into the distribution of the different flowers in parameter space: notably, iris setosa is much more distinct than the other two species.
Note also that the default implementation of PCA computes the
singular value decomposition (SVD) of the full
data matrix, which is not scalable when both n_samples
and
n_features
are big (more that a few thousands).
If you are interested in a number of components that is much
smaller than both n_samples
and n_features
, consider using
sklearn.decomposition.RandomizedPCA
instead.
Other dimensionality reduction techniques which are useful to know about:
sklearn.decomposition.PCA
: sklearn.decomposition.RandomizedPCA
:sklearn.decomposition.SparsePCA
:sklearn.decomposition.FastICA
:sklearn.decomposition.NMF
:sklearn.manifold.LocallyLinearEmbedding
: sklearn.manifold.IsoMap
: Repeat the above dimensionality reduction with
sklearn.decomposition.RandomizedPCA
.
You can re-use the plot_PCA_2D
function from above.
Are the results similar to those from standard PCA?