Scikit-Learn has a good datasets submodule that you could load a variety of datasets from. Especially, for beginning if you want to play or apply basic machine learning algorithms, you could still immediately start playing around with datasets, which we will do exactly now!
%matplotlib inline
import numpy as np
import matplotlib as mlp
import matplotlib.pyplot as plt
# For 3-D swiss roll
from mpl_toolkits.mplot3d import Axes3D
import scipy
import pandas as pd
from sklearn import datasets
Do not include the "hidden" datasets
available_datasets = [ii for ii in dir(datasets) if not ii.startswith('_')]
available_datasets
['base', 'california_housing', 'clear_data_home', 'covtype', 'dump_svmlight_file', 'fetch_20newsgroups', 'fetch_20newsgroups_vectorized', 'fetch_california_housing', 'fetch_covtype', 'fetch_lfw_pairs', 'fetch_lfw_people', 'fetch_mldata', 'fetch_olivetti_faces', 'fetch_species_distributions', 'get_data_home', 'lfw', 'load_boston', 'load_diabetes', 'load_digits', 'load_files', 'load_iris', 'load_lfw_pairs', 'load_lfw_people', 'load_linnerud', 'load_mlcomp', 'load_sample_image', 'load_sample_images', 'load_svmlight_file', 'load_svmlight_files', 'make_biclusters', 'make_blobs', 'make_checkerboard', 'make_circles', 'make_classification', 'make_friedman1', 'make_friedman2', 'make_friedman3', 'make_gaussian_quantiles', 'make_hastie_10_2', 'make_low_rank_matrix', 'make_moons', 'make_multilabel_classification', 'make_regression', 'make_s_curve', 'make_sparse_coded_signal', 'make_sparse_spd_matrix', 'make_sparse_uncorrelated', 'make_spd_matrix', 'make_swiss_roll', 'mlcomp', 'mldata', 'mldata_filename', 'olivetti_faces', 'samples_generator', 'species_distributions', 'svmlight_format', 'twenty_newsgroups']
Let's get a dataset:
swiss_roll, color = datasets.samples_generator.make_swiss_roll(n_samples=2500)
fig, ax = plt.subplots(figsize=(16, 12))
ax.scatter(swiss_roll[:, 0], swiss_roll[:, 2], c=color, cmap=mlp.cm.Spectral)
ax.set_title("Swiss Roll");
plt.axis('tight');
plt.xticks([]); plt.yticks([]);
n_samples
parameter to a large number and see what happens? What about when you decrease it?n_samples
=100 ? In general, data may not be abundant. Can you think any methods to deal with missing or undersampled data?digits = datasets.load_digits()
type(digits)
sklearn.datasets.base.Bunch
Bunch is a dictionary like data structure, we could look at which fields it has by simply looking at keys
function like we do
in dictionaries. It provides a nice way to access the fields by using bunch.field
where dictionary needs dict['field']
. If you are familiar with Javascript, it resembles a lot with Javascript object. However, dictionary like acess still works, which I will use like a dictionary.
digits.keys()
['images', 'data', 'target_names', 'DESCR', 'target']
Let's see what it is
print(digits['DESCR'])
Optical Recognition of Handwritten Digits Data Set Notes ----- Data Set Characteristics: :Number of Instances: 5620 :Number of Attributes: 64 :Attribute Information: 8x8 image of integer pixels in the range 0..16. :Missing Attribute Values: None :Creator: E. Alpaydin (alpaydin '@' boun.edu.tr) :Date: July; 1998 This is a copy of the test set of the UCI ML hand-written digits datasets http://archive.ics.uci.edu/ml/datasets/Optical+Recognition+of+Handwritten+Digits The data set contains images of hand-written digits: 10 classes where each class refers to a digit. Preprocessing programs made available by NIST were used to extract normalized bitmaps of handwritten digits from a preprinted form. From a total of 43 people, 30 contributed to the training set and different 13 to the test set. 32x32 bitmaps are divided into nonoverlapping blocks of 4x4 and the number of on pixels are counted in each block. This generates an input matrix of 8x8 where each element is an integer in the range 0..16. This reduces dimensionality and gives invariance to small distortions. For info on NIST preprocessing routines, see M. D. Garris, J. L. Blue, G. T. Candela, D. L. Dimmick, J. Geist, P. J. Grother, S. A. Janet, and C. L. Wilson, NIST Form-Based Handprint Recognition System, NISTIR 5469, 1994. References ---------- - C. Kaynak (1995) Methods of Combining Multiple Classifiers and Their Applications to Handwritten Digit Recognition, MSc Thesis, Institute of Graduate Studies in Science and Engineering, Bogazici University. - E. Alpaydin, C. Kaynak (1998) Cascading Classifiers, Kybernetika. - Ken Tang and Ponnuthurai N. Suganthan and Xi Yao and A. Kai Qin. Linear dimensionalityreduction using relevance weighted LDA. School of Electrical and Electronic Engineering Nanyang Technological University. 2005. - Claudio Gentile. A New Approximate Maximal Margin Classification Algorithm. NIPS. 2000.
digits_data = digits['images']
digits_data.shape
(1797, 8, 8)
Let's look at the dimensions of the digits
plt.matshow(digits_data[0], interpolation='nearest', cmap=plt.cm.gray);
1797 instances with images sized 8x8.
fig = plt.figure(figsize=(16, 16))
for ii in range(64):
plt.subplot(8, 8, ii + 1) # It starts with one
plt.imshow(digits_data[ii], cmap=plt.cm.gray)
plt.grid(False);
plt.xticks([]);
plt.yticks([]);
faces = datasets.olivetti_faces.fetch_olivetti_faces()
faces.keys()
['images', 'data', 'target', 'DESCR']
faces_data = faces['images']
faces_data.shape
(400, 64, 64)
There are 400 face observations which are sized 64x64.
fig = plt.figure(figsize=(16, 16))
for ii in range(64):
plt.subplot(8, 8, ii + 1) # It starts with one
plt.imshow(faces_data[ii], cmap=plt.cm.gray)
plt.grid(False);
plt.xticks([]);
plt.yticks([]);
california_housing = datasets.fetch_california_housing()
california_housing.keys()
['data', 'feature_names', 'DESCR', 'target']
print(california_housing.DESCR)
California housing dataset. The original database is available from StatLib http://lib.stat.cmu.edu/ The data contains 20,640 observations on 9 variables. This dataset contains the average house value as target variable and the following input variables (features): average income, housing average age, average rooms, average bedrooms, population, average occupation, latitude, and longitude in that order. References ---------- Pace, R. Kelley and Ronald Barry, Sparse Spatial Autoregressions, Statistics and Probability Letters, 33 (1997) 291-297.
california_data = california_housing.data
california_data.shape
(20640, 8)
Can you explore(maybe visualize) the dataset in a similar way?