The IPython shell

  • IPython is an enhanced Python shell for interactive computing
  • Provides facilities for distributed and parallel computing
  • Allows interactive evaluation of results
In [1]:
from math import pi 
r = 3.
2 * pi * r

This flexible style matches well the spirit of computing in a scientific context, in which determining what computations must be performed next often requires significant work. An interactive environment lets scientists look at data, test new ideas, combine algorithmic approaches, and evaluate their outcomes directly.

Pérez and Granger, IPython: A System for Interactive Scientific Computing

IPython is an enhanced Python shell for interactive distributed and parallel computing. When users are testing new ideas and algorithms in computer vision, evaluating the results directly, in an interactive way, is more convenient than the traditional compile-then-execute cycle. A live computing state is convenient because previous intermediate results (images, feature vectors, parameters) are kept available for exploration. Pérez and Granger argue that, in a research context, determining which computations must be performed next often requires significant work.


  • Access to all session state
  • TAB completion
  • System shell integration
  • Dynamic introspection and help
    • Direct manipulation of objects in memory
  • Magic commands (secondary control system)

Session state

In [2]:
2 * _1
In [3]:
In [4]:

In an interactive computing system, users should have access to all session state. In IPython, this access is not just provided by the dynamically attributed variables, but also by numbered output prompts. Previous computations can be retrieved using and underscore "_" and the number of the output.

System shell integration

Other useful capabilities in IPython are TAB completion and system shell integration. When the user types the TAB key, IPython tries to complete the current prompt with keywords or the names of methods, variables and files in the current directory. This is particularly useful when exploring unfamiliar or large APIs as in OpenCV or in the SciPy library. Regarding system shell integration, IPython cannot only call any system command, but also can capture the shell’s output in Python variables and call system commands with values computed from variables.

Call a system's command:

In [5]:
ls ../data/*.jpg
ls: não é possível acessar '../data/*.jpg': Arquivo ou diretório não encontrado

Store the command output in a Python variable:

In [6]:
jpgs_list = !ls data/*.jpg

Call a system's command passing arguments stored in Python variables:

In [7]:
fname = jpgs_list[0]
!du -h $fname
!ls -l $fname
20K	data/BSD-118035.jpg
-rw-r--r-- 1 thiago thiago 18556 Fev 10 14:36 data/BSD-118035.jpg

Magic functions

  • IPython presents 98 magic functions
  • They are a secondary command system, providing extra capabilities
In [8]:

Some examples

  • %load and %save - load/save code into/from IPython
  • %run- run a Python file inside IPython as a program
  • %who - list all interactive variables
  • %timeit - time execution of a Python statement or expression


In [9]:
fname	 jpgs_list	 pi	 r	 
In [10]:
%who str
In [11]:
%who float
pi	 r	 


In [12]:
from random import random
%timeit random()
The slowest run took 106.32 times longer than the fastest. This could mean that an intermediate result is being cached.
10000000 loops, best of 3: 47.1 ns per loop
In [13]:
# Import the linear algebra module in SciPy
from scipy import linalg

Create a random $100 \times 100$ matrix $A$ and time the Singular Value Decompostion (SVD) $A = U\Sigma V^\intercal$.

In [14]:
from numpy.random import randn
In [15]:
%%timeit A = randn(10000).reshape(100,100)
100 loops, best of 3: 8.47 ms per loop

Timing 2D features detection (SIFT vs. SURF)

In [16]:
import cv2
lenna = cv2.imread('data/lenna.tiff', cv2.IMREAD_GRAYSCALE)
In [17]:
sift = cv2.xfeatures2d.SIFT_create()
%timeit sift.detectAndCompute(lenna, None)
10 loops, best of 3: 150 ms per loop
In [18]:
surf = cv2.xfeatures2d.SURF_create()
%timeit surf.detectAndCompute(lenna, None)
10 loops, best of 3: 122 ms per loop

The IPython notebook

  • Web-based version of the IPython shell
  • Chunks of text and code are organized in cells
  • Cells can be inserted, deleted, rearranged and executed as needed
  • Notebooks can be converted to Python programs, PDF documents and slides
    • These slides and the tutorial's handouts are notebooks convertions!
  • IPython notebooks are executable documents

The IPython Notebook is a web-based version of the IPython shell presenting extended functionality. In the notebook, the user can organize formatted text and code blocks in a flexible way. Text and code are organized in cells that can be inserted, deleted, rearranged and executed as needed. IPython notebooks can handle plots, mathematical formulas and code output, everything organized in a single executable document. Notebooks are being used for research notes, and on the production of articles and books.

Markdown cells

In a notebook, a Markdown cell is able the render rich-formatted text using the Markdown markup convention. Mathematical notation is defined using LaTeX syntax and rendered by MathJax.

Let $\mu_I$ and $\sigma_II$ be the mean and the standard deviation of a grayscale 
image $I$. The whitening operation is defined by:

$W_I[i,j] = \frac{I[i,j] - \mu_I}{\sigma_I}$.

Let $\mu_I$ and $\sigma_II$ be the mean and the standard deviation of a grayscale image $I$. The whitening operation is defined by:
$W_I[i,j] = \frac{I[i,j] - \mu_I}{\sigma_I}$.

Code cells

  • Code cells keep (short) Python scripts
  • The browser sends the code to the server running an IPython Kernel
  • Output is sent back to the browser for exhibition
In [19]:
hello = "Hello, World! I'm a string in an IPython notebook code cell."
print hello
Hello, World! I'm a string in an IPython notebook code cell.
In [20]:

Code cells contain Python code that is sent to the IPython interpreter running on the server, executed, and the resulting output sent back to the browser for exhibition. That means that IPython can be running in a powerful machine like a server while the user is able to perform her work from a leaner system as a laptop or a tablet. If the result of a code block is a plot, it can be exhibited inside the notebook in the browser (inline plotting).

Notebook files (.ipynb)

  • IPython notebooks are stored as JSON files
  • Can be converted to:

    • a Python script
    • a reStructuredText document
    • a HTML document
    • a LaTeX document
    • a Markdown document
    • slides
  • For details, check:

$ jupyter nbconvert --help

IPython notebooks are stored in JSON files that keep the cells’ content. These files uses the extension .ipynb and can be exported as Python scripts, HTML documents or even printed. But what makes notebook files suitable to reproducible research is the fact they are executable documents, not only able to store textual and mathematical descriptions but also replicate the computations.

Starting IPython

The IPython default console (terminal) can be started using:

$ ipython 
Python 2.7.6 (default, Mar 22 2014, 22:59:56) 
Type "copyright", "credits" or "license" for more information.

IPython 2.1.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: 

Other alternative is qtconsole, a console running on a graphical window:

$ ipython qtconsole

To run notebooks, use jupyter:

$ jupyter notebook
[NotebookApp] Using existing profile dir: u'/home/thiago/.ipython/profile_default'
[NotebookApp] Using MathJax from CDN:
[NotebookApp] Serving notebooks from local directory: /home/thiago
[NotebookApp] 0 active kernels 
[NotebookApp] The IPython Notebook is running at: http://localhost:8888/
[NotebookApp] Use Control-C to stop this server and shut down all 
kernels (twice to skip confirmation).
  • The URL to access the served notebooks is http://localhost:8888
  • System is serving the notebooks in the directory /home/thiago
  • User can employ Ctrl-C to stop the server and shutdown kernels
  • Jupyter supports IPython, R and Julia kernels. More languages coming!