Notebook

Plotting and visualizing data Chapter 1.6¶

Turns out that plotting is a recent invention! One of the pioneers of this, well, art was a british economist, Playfair, whose work dates to the mid 1700. The first time series, at least the first ones that we can understand and follow. Until then people were just reporting numbers in the forms of tables. Of course there are several advantages in plotting, over just showing tables: We resort to¶

- plotting to summarize the data¶

- plotting to encourage comparison (time series)¶

And if we choose to talk about Data Mining, and Machine Learning it is because our data is large (how large is large??) so visualization becomes necesary, and not just useful. This shift happened in astronomy decades ago. The biggest name in the visualization community is Tufte, who was the first -that I know of- and certainly the most famous visualization scholar. I am bringing his books for you to browse, and he is mentioned in the SDMMLA book. I will also mention Jer Thorp, who i brought to NYU for a seminar, as a more modern "data artist" (his choice of words).¶

Following Tufte, and Thorp, and really anyone else interested in visualization, as far back as Playfair, who used to write very interesting bits to go with his figures, we highlight 2 aspects to visualization. We visualize in order to:¶

- present our findings: "explanatory visualization"¶

- discover patterns in our data:"exploratory visualization"¶

Other terms commonly used by Thorp is "reduction/revelation" graphics. Tufte is mostly focussed on the earlier. And it seems to me that so are the authors of this book. But as we increase the size of the dataset new relations will appear in dimensions that are, to our knowledge, not connected yet. To know which we can either blindly try or let visualizations guide our exploration compass!¶

examples of explainatory visualizations:¶

In [1]:

from IPython.display import Image
%pylab inline
i = Image(url="http://cosmo.nyu.edu/~fb55/data/imgres-1.jpg", width=1500)
display(i)

Welcome to pylab, a matplotlib-based Python environment [backend: module://IPython.kernel.zmq.pylab.backend_inline].
For more information, type 'help(pylab)'.

Early time series in Playfairs'work: the price of Weat in GB over time (histogram), and the wages of labour (red line).

In [13]:

i = Image(url="http://cosmo.nyu.edu/~fb55/data/fig1.jpg", width=1300)
display(i)

From Smith, Mauerhan, Prieto, 2013, these time series encourage comparison, and reveal striking similarities between SN impostors.

In [8]:

i = Image(url="http://cosmo.nyu.edu/~fb55/data/imgres.jpg", width=1500)
display(i)

First ever pie chart. Pie chartes were invented with this plot by Playwrite the explanation that went with it says: "we have a more accurate idea of the sizes of the planets, which are spheres, than of the nations of Europe which we see on the maps, all of which are irregular forms in themselves as well as unlike to each other"

examples of exploratory visualizations¶

In [24]:

i = Image(url="http://cosmo.nyu.edu/~fb55/data/lickgals.jpg", width=1000)
display(i)

Sendler et al, 1977 "New Reduction of the Lick Catalog of Galaxies" shows large scale structure for the first time

Beware of course of the bias the eye has in detecting patterns and always validate their statistical significance! as pointed out indeed by Sendler et al 1977.¶

Now, how to visualize, and what to visualize? That is both an aesthetic and a scientific problem. You need the reader to be engaged by the graphic, but not distracted by it, and as data increases the risk of plotting too much information increases as well. I heard Alyssa Goodman say multiple times that a human can understand 7 different modes of information at one time, but not more. SDMMLA presents some guidelines, which i am paraphrasing as¶

-clarity¶

-coherence¶

-proper scaling¶

-proper relations¶

From chapter 1.2 we decided this book (and in general Data Mining) is about estimating the distribution of h(x), a physical quantity from which we draw x (from f(x), the empirical distribution). The most intuitive representation of this is a scatter plot.¶

In [10]:

# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
from matplotlib import pyplot as plt
from astroML.datasets import fetch_moving_objects

#----------------------------------------------------------------------
# This function adjusts matplotlib settings for a uniform feel in the textbook.
# Note that with usetex=True, fonts are rendered with LaTeX.  This may
# result in an error if LaTeX is not installed on your system.  In that case,
# you can set usetex to False.
from astroML.plotting import setup_text_plots
setup_text_plots(fontsize=8, usetex=False)
%pylab inline
#------------------------------------------------------------
# Fetch the moving object data
data = fetch_moving_objects(Parker2008_cuts=True)

# Use only the first 10000 points
data = data[:10000]

a = data['aprime']
sini = data['sin_iprime']

#------------------------------------------------------------
# Plot the results
fig, ax = plt.subplots(figsize=(10,7.5))
ax.plot(a, sini, '.', markersize=2, color='black')

ax.set_xlim(2.0, 3.6)
ax.set_ylim(-0.01, 0.31)

ax.set_xlabel('Semimajor Axis (AU)')
ax.set_ylabel('Sine of Inclination Angle')

plt.show()

Welcome to pylab, a matplotlib-based Python environment [backend: module://IPython.kernel.zmq.pylab.backend_inline].
For more information, type 'help(pylab)'.

Figure 1.8: A simple scatter plot is perfect for these data, in this space: semimajor axis vs inclination (sin(i)) of small solar system objects in the middle solar system. It reveals overdensities, underdensitiesand the gaps, that are real and physical - the gaps caused by the resonances with Jupiter. That is because the number and the clustering of the points is just right, so no brainer how to plot this (excellent for both exploratory and explanatory purposes).

In [21]:

# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
from matplotlib import pyplot as plt

from astroML.plotting import scatter_contour
from astroML.datasets import fetch_sdss_S82standards

#----------------------------------------------------------------------
# This function adjusts matplotlib settings for a uniform feel in the textbook.
# Note that with usetex=True, fonts are rendered with LaTeX.  This may
# result in an error if LaTeX is not installed on your system.  In that case,
# you can set usetex to False.
from astroML.plotting import setup_text_plots
setup_text_plots(fontsize=8, usetex=False)

#------------------------------------------------------------
# Fetch the Stripe 82 standard star catalog

data = fetch_sdss_S82standards()

g = data['mmu_g']
r = data['mmu_r']
i = data['mmu_i']

#------------------------------------------------------------
# plot the results
fig, ax = plt.subplots(figsize=(10,7.5))
ax.plot(g-r,r-i, '.', markersize=2, color='black')

ax.set_xlabel(r'${\rm g - r}$')
ax.set_ylabel(r'${\rm r - i}$')

ax.set_xlim(-0.6, 2.5)
ax.set_ylim(-0.6, 2.5)

plt.show()

Not so much for all SDSS standard stars (subset in the SDMMLA) color-color: here the density in the core region is too high, and too low in the ourskirts. it fails on clarity, proper scaling, and proper relation metrics. How to reveal both the structure in the high density and in the low density region? there are ways, that generally imply choosing thresholds or stretch. But it is important to enphasize that when we make a threshold choice, as we are about to do, and as we ALWAYS do in histograms (Chapter 6), the choice of threshold MUST BE STATISTICALLY APPROACHED! Or we are bound to give a biassed and possibly incorrect visualization of the data.

In [19]:

# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
from matplotlib import pyplot as plt

from astroML.plotting import scatter_contour
from astroML.datasets import fetch_sdss_S82standards

#----------------------------------------------------------------------
# This function adjusts matplotlib settings for a uniform feel in the textbook.
# Note that with usetex=True, fonts are rendered with LaTeX.  This may
# result in an error if LaTeX is not installed on your system.  In that case,
# you can set usetex to False.
from astroML.plotting import setup_text_plots
setup_text_plots(fontsize=8, usetex=False)

#------------------------------------------------------------
# Fetch the Stripe 82 standard star catalog

data = fetch_sdss_S82standards()

g = data['mmu_g']
r = data['mmu_r']
i = data['mmu_i']

#------------------------------------------------------------
# plot the results
fig, ax = plt.subplots(figsize=(10,7.5))
scatter_contour(g - r, r - i, threshold=200, log_counts=True, ax=ax,
                histogram2d_args=dict(bins=40),
                plot_args=dict(marker=',', linestyle='none', color='black'),
                contour_args=dict(cmap=plt.cm.bone))

ax.set_xlabel(r'${\rm g - r}$')
ax.set_ylabel(r'${\rm r - i}$')

ax.set_xlim(-0.6, 2.5)
ax.set_ylim(-0.6, 2.5)

plt.show()

Figure 1.9, scatter contour plot

In [18]:

# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
import numpy as np
from matplotlib import pyplot as plt

from astroML.datasets import fetch_sdss_S82standards

#----------------------------------------------------------------------
# This function adjusts matplotlib settings for a uniform feel in the textbook.
# Note that with usetex=True, fonts are rendered with LaTeX.  This may
# result in an error if LaTeX is not installed on your system.  In that case,
# you can set usetex to False.
from astroML.plotting import setup_text_plots
setup_text_plots(fontsize=8, usetex=False)

#------------------------------------------------------------
# Fetch the stripe 82 data
data = fetch_sdss_S82standards()

g = data['mmu_g']
r = data['mmu_r']
i = data['mmu_i']

#------------------------------------------------------------
# Compute and plot the 2D histogram
H, xbins, ybins = np.histogram2d(g - r, r - i,
                                 bins=(np.linspace(-0.5, 2.5, 50),
                                       np.linspace(-0.5, 2.5, 50)))

# Create a black and white color map where bad data (NaNs) are white
cmap = plt.cm.binary
cmap.set_bad('w', 1.)

# Use the image display function imshow() to plot the result
fig, ax = plt.subplots(figsize=(10,7.5))
H[H == 0] = 1  # prevent warnings in log10
ax.imshow(np.log10(H).T, origin='lower',
          extent=[xbins[0], xbins[-1], ybins[0], ybins[-1]],
          cmap=cmap, interpolation='nearest',
          aspect='auto')

ax.set_xlabel(r'${\rm g - r}$')
ax.set_ylabel(r'${\rm r - i}$')

ax.set_xlim(-0.6, 2.5)
ax.set_ylim(-0.6, 2.5)

plt.show()

Figure 1.10 Hess plot (binning data with a 2D histogram where color in each bin represent density)

What about colors?? I think colors are awesome. They enable visualization of a third dimension and relating different plots (Figure 1.12). Higher dimensionality representation is also an important topic. All the same I would say the community is still devided on the usefulness and appropriateness of color plots. Personally I am all for it. We can invest in color printers and look at plots online!¶

In [11]:

# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
import numpy as np
from matplotlib import pyplot as plt
from astroML.datasets import fetch_moving_objects
from astroML.plotting.tools import devectorize_axes

#----------------------------------------------------------------------
# This function adjusts matplotlib settings for a uniform feel in the textbook.
# Note that with usetex=True, fonts are rendered with LaTeX.  This may
# result in an error if LaTeX is not installed on your system.  In that case,
# you can set usetex to False.
from astroML.plotting import setup_text_plots
setup_text_plots(fontsize=8, usetex=False)


def black_bg_subplot(*args, **kwargs):
    """Create a subplot with black background"""
    kwargs['axisbg'] = 'k'
    ax = plt.subplot(*args, **kwargs)

    # set ticks and labels to white
    for spine in ax.spines.values():
        spine.set_color('w')

    for tick in ax.xaxis.get_major_ticks() + ax.yaxis.get_major_ticks():
        for child in tick.get_children():
            child.set_color('w')

    return ax


def compute_color(mag_a, mag_i, mag_z, a_crit=-0.1):
    """
    Compute the scatter-plot color using code adapted from
    TCL source used in Parker 2008.
    """
    # define the base color scalings
    R = np.ones_like(mag_i)
    G = 0.5 * 10 ** (-2 * (mag_i - mag_z - 0.01))
    B = 1.5 * 10 ** (-8 * (mag_a + 0.0))

    # enhance green beyond the a_crit cutoff
    G += 10. / (1 + np.exp((mag_a - a_crit) / 0.02))

    # normalize color of each point to its maximum component
    RGB = np.vstack([R, G, B])
    RGB /= RGB.max(0)

    # return an array of RGB colors, which is shape (n_points, 3)
    return RGB.T


#------------------------------------------------------------
# Fetch data and extract the desired quantities
data = fetch_moving_objects(Parker2008_cuts=True)
mag_a = data['mag_a']
mag_i = data['mag_i']
mag_z = data['mag_z']
a = data['aprime']
sini = data['sin_iprime']

# dither: magnitudes are recorded only to +/- 0.01
mag_a += -0.005 + 0.01 * np.random.random(size=mag_a.shape)
mag_i += -0.005 + 0.01 * np.random.random(size=mag_i.shape)
mag_z += -0.005 + 0.01 * np.random.random(size=mag_z.shape)

# compute RGB color based on magnitudes
color = compute_color(mag_a, mag_i, mag_z)

#------------------------------------------------------------
# set up the plot
fig = plt.figure(figsize=(20,7.7), facecolor='k')
fig.subplots_adjust(left=0.1, right=0.95, wspace=0.3,
                    bottom=0.2, top=0.93)

# plot the color-magnitude plot
ax = black_bg_subplot(121)
ax.scatter(mag_a, mag_i - mag_z,
           c=color, s=0.5, lw=0)
devectorize_axes(ax, dpi=400)

ax.plot([0, 0], [-0.8, 0.6], '--w', lw=1)
ax.plot([0, 0.4], [-0.15, -0.15], '--w', lw=1)

ax.set_xlim(-0.3, 0.4)
ax.set_ylim(-0.8, 0.6)

ax.set_xlabel(r'${\rm a*}$', color='w')
ax.set_ylabel(r'${\rm i-z}$', color='w')

# plot the orbital parameters plot
ax = black_bg_subplot(122)
ax.scatter(a, sini,
           c=color, s=0.5, lw=0, edgecolor='none')
devectorize_axes(ax, dpi=400)

ax.plot([2.5, 2.5], [-0.02, 0.3], '--w', lw=1)
ax.plot([2.82, 2.82], [-0.02, 0.3], '--w', lw=1)

ax.set_xlim(2.0, 3.3)
ax.set_ylim(-0.02, 0.3)

ax.set_xlabel(r'${\rm a (AU)}$', color='w')
ax.set_ylabel(r'${\rm sin(i)}$', color='w')

# label the plot
text_kwargs = dict(color='w', transform=plt.gca().transAxes,
                   ha='center', va='bottom')

ax.text(0.25, 1.02, 'Inner', **text_kwargs)
ax.text(0.53, 1.02, 'Mid', **text_kwargs)
ax.text(0.83, 1.02, 'Outer', **text_kwargs)

# Saving the black-background figure requires some extra arguments:
#fig.savefig('moving_objects.png',
#            facecolor='black',
#            edgecolor='none')

plt.show()

A third dimension as well as a relation between two visualizations is expressed by the colors.