IPython: an enviromnent for interactive computing

What is IPython?

• Short for Interactive Python
• A platform for you to interact with your code and data
• The notebook: a system for literate computing
  • The combination of narrative, code and results
  • Weave your scientific narratives together with your computational process
• Tools for easy parallel computing
  • Interact with many processes

IPython at the terminal

The basic IPython client: at the terminal, simply type ipython:

$ ipython
Python 3.4.3 (default, Feb 24 2015, 22:44:40) 
Type "copyright", "credits" or "license" for more information.

IPython 3.1.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: print("hello world")
hello world

In the notebook use the help menu, then about:

Server Information:

You are using Jupyter notebook.

    The version of the notebook server is 5.0.0.dev-21a6aec and is running on:
    Python 3.5.2 |Anaconda custom (x86_64)| (default, Jul  2 2016, 17:52:12) 
    [GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)]

Current Kernel Information:

    Python 3.5.2 |Anaconda custom (x86_64)| (default, Jul  2 2016, 17:52:12) 
    Type "copyright", "credits" or "license" for more information.

    IPython 5.2.0.dev -- An enhanced Interactive Python.
    ?         -> Introduction and overview of IPython's features.
    %quickref -> Quick reference.
    help      -> Python's own help system.
    object?   -> Details about 'object', use 'object??' for extra details.

Some other tutorial help/resources :

• The Jupyter website
• Search for "IPython in depth" tutorial on youtube and pyvideo, much longer, much deeper
• Ask for help on Stackoverflow, tag it "ipython" or "Jupyter"
• Mailing list
• File a github issue
• Twitter
• Reddit
• Notebook Gallery
  • full books
• Come talk to the team! We're In BIDS !

IPython: beyond plain Python

When executing code in IPython, all valid Python syntax works as-is, but IPython provides a number of features designed to make the interactive experience more fluid and efficient.

First things first: running code, getting help

In the notebook, to run a cell of code, hit Shift-Enter. This executes the cell and puts the cursor in the next cell below, or makes a new one if you are at the end. Alternately, you can use:

• Alt-Enter to force the creation of a new cell unconditionally (useful when inserting new content in the middle of an existing notebook).
• Control-Enter executes the cell and keeps the cursor in the same cell, useful for quick experimentation of snippets that you don't need to keep permanently.

print("Hello")

Hello

Getting help

?

Help with ? and ??

Typing object_name? will print all sorts of details about any object, including docstrings, function definition lines (for call arguments) and constructor details for classes.

import collections
collections.namedtuple?

collections.Counter??

*int*?

An IPython quick reference card:

%quickref

Tab completion

Tab completion, especially for attributes, is a convenient way to explore the structure of any object you’re dealing with. Simply type object_name.<TAB> to view the object’s attributes. Besides Python objects and keywords, tab completion also works on file and directory names.

collections. #<use tab>

  File "<ipython-input-7-a0f893a4c6a3>", line 1
    collections. #<use tab>
                           ^
SyntaxError: invalid syntax

collections.namedtuple( #<use shift tab>

  File "<ipython-input-8-372f83d3498d>", line 1
    collections.namedtuple( #<use shift tab>
                                            ^
SyntaxError: unexpected EOF while parsing

The interactive workflow: input, output, history

2 + 10

12

_ + 10

22

Output control

You can suppress the storage and rendering of output if you append ; to the last cell (this comes in handy when plotting with matplotlib, for example):

10+20;

_

22

Output history

The output is stored in _N and Out[N] variables:

#This number (11) may be change, depending on the execution number of the cell above.
_11 == Out[11]

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-13-2aca1487d4be> in <module>()
      1 #This number (11) may be change, depending on the execution number of the cell above.
----> 2 _11 == Out[11]

NameError: name '_11' is not defined

Out

{9: 12, 10: 22, 12: 22}

And the last three have shorthands for convenience:

print('last output:', _)
print('next one   :', __)
print('and next   :', ___)

last output: 22
next one   : 22
and next   : 12

Output vs Display

1+1

2

print(1+1)

2

Can you spot the difference in the 2 above cells ?

The input history is also available

In[11]

'10+20;'

_i

'In[11]'

_ii

'In[11]'

print('last input:', _i)
print('next one  :', _ii)
print('and next  :', _iii)

last input: _ii
next one  : _i
and next  : In[11]

%history

print("Hello")
?
import collections
collections.namedtuple?
collections.Counter??
*int*?
%quickref
collections. #<use tab>
collections.namedtuple( #<use shift tab>
2 + 10
_ + 10
10+20;
_
#This number (11) may be change, depending on the execution number of the cell above.
_11 == Out[11]
Out
print('last output:', _)
print('next one   :', __)
print('and next   :', ___)
1+1
print(1+1)
In[11]
_i
_ii
print('last input:', _i)
print('next one  :', _ii)
print('and next  :', _iii)
%history

Accessing the underlying operating system

Note: the commands below work on Linux or Macs, but may behave differently on Windows, as the underlying OS is different. IPython's ability to access the OS is still the same, it's just the syntax that varies per OS.

#For Windows users, change to !dir
!pwd

/Users/bussonniermatthias/dev/python-bootcamp/Lectures/04_IPythonNotebookIntroduction

files = !ls
print("My current directory's files:")
print(files)

My current directory's files:
['Exercises.ipynb', 'IPython - beyond plain Python.ipynb', 'Markdown Cells.ipynb', 'Notebook Basics.ipynb', 'README.md', 'animation.m4v', 'ipython-book.png', 'mknbindex.py', 'python-logo.svg', 'style.css', 'talktools.py']

!echo $files

[Exercises.ipynb, IPython - beyond plain Python.ipynb, Markdown Cells.ipynb, Notebook Basics.ipynb, README.md, animation.m4v, ipython-book.png, mknbindex.py, python-logo.svg, style.css, talktools.py]

!echo {files[0].upper()}

EXERCISES.IPYNB

Beyond Python: magic functions

The IPyhton 'magic' functions are a set of commands, invoked by prepending one or two % signs to their name, that live in a namespace separate from your normal Python variables and provide a more command-like interface. They take flags with -- and arguments without quotes, parentheses or commas. The motivation behind this system is two-fold:

• To provide an orthogonal namespace for controlling IPython itself and exposing other system-oriented functionality.

• To expose a calling mode that requires minimal verbosity and typing while working interactively. Thus the inspiration taken from the classic Unix shell style for commands.

%magic

Line vs cell magics:

%timeit range(10)

1000000 loops, best of 3: 292 ns per loop

%%timeit
range(10)
range(100)

1000000 loops, best of 3: 588 ns per loop

Line magics can be used even inside code blocks:

import sys
results = []
for i in range(5):
    size = i*10
    print('size:',size) 
    result = %timeit -o list(range(size))
    results.append(result)
    
results

size: 0
The slowest run took 4.83 times longer than the fastest. This could mean that an intermediate result is being cached.
1000000 loops, best of 3: 402 ns per loop
size: 10
1000000 loops, best of 3: 661 ns per loop
size: 20
1000000 loops, best of 3: 762 ns per loop
size: 30
1000000 loops, best of 3: 899 ns per loop
size: 40
1000000 loops, best of 3: 923 ns per loop

[<TimeitResult : 1000000 loops, best of 3: 402 ns per loop>,
 <TimeitResult : 1000000 loops, best of 3: 661 ns per loop>,
 <TimeitResult : 1000000 loops, best of 3: 762 ns per loop>,
 <TimeitResult : 1000000 loops, best of 3: 899 ns per loop>,
 <TimeitResult : 1000000 loops, best of 3: 923 ns per loop>]

Magics can do anything they want with their input, so it doesn't have to be valid Python:

%%bash
echo "My shell is:" $SHELL
echo "My memory status is:" free

My shell is: /bin/bash
My memory status is: free

Another interesting cell magic: create any file you want locally from the notebook:

%%writefile test.txt
This is a test file!
It can contain anything I want...

more...

Writing test.txt

!cat test.txt

This is a test file!
It can contain anything I want...

more...

Let's see what other magics are currently defined in the system:

%lsmagic

Available line magics:
%alias  %alias_magic  %autocall  %automagic  %autosave  %bookmark  %cat  %cd  %clear  %colors  %config  %connect_info  %cp  %debug  %dhist  %dirs  %doctest_mode  %ed  %edit  %env  %gui  %hist  %history  %killbgscripts  %ldir  %less  %lf  %lk  %ll  %load  %load_ext  %loadpy  %logoff  %logon  %logstart  %logstate  %logstop  %ls  %lsmagic  %lx  %macro  %magic  %man  %matplotlib  %mkdir  %more  %mv  %notebook  %page  %pastebin  %pdb  %pdef  %pdoc  %pfile  %pinfo  %pinfo2  %popd  %pprint  %precision  %profile  %prun  %psearch  %psource  %pushd  %pwd  %pycat  %pylab  %qtconsole  %quickref  %recall  %rehashx  %reload_ext  %rep  %rerun  %reset  %reset_selective  %rm  %rmdir  %run  %save  %sc  %set_env  %store  %sx  %system  %tb  %time  %timeit  %unalias  %unload_ext  %who  %who_ls  %whos  %xdel  %xmode

Available cell magics:
%%!  %%HTML  %%SVG  %%bash  %%capture  %%debug  %%file  %%html  %%javascript  %%js  %%latex  %%perl  %%prun  %%pypy  %%python  %%python2  %%python3  %%ruby  %%script  %%sh  %%svg  %%sx  %%system  %%time  %%timeit  %%writefile

Automagic is ON, % prefix IS NOT needed for line magics.

Running normal Python code: execution and errors

Not only can you input normal Python code, you can even paste straight from a Python or IPython shell session:

>>> # Fibonacci series:
... # the sum of two elements defines the next
... a, b = 0, 1
>>> while b < 10:
...     print(b)
...     a, b = b, a+b

1
1
2
3
5
8

In [1]: for i in range(10):
   ...:     print(i),
   ...:     

0
1
2
3
4
5
6
7
8
9

Error display

And when your code produces errors, you can control how they are displayed with the %xmode magic:

%%writefile mod.py

def f(x):
    return 1.0/(x-1)

def g(y):
    return f(y+1)

Writing mod.py

Now let's call the function g with an argument that would produce an error:

import mod
mod.g(0)

---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
<ipython-input-38-a54c5799f57e> in <module>()
      1 import mod
----> 2 mod.g(0)

/Users/bussonniermatthias/dev/python-bootcamp/Lectures/04_IPythonNotebookIntroduction/mod.py in g(y)
      4 
      5 def g(y):
----> 6     return f(y+1)

/Users/bussonniermatthias/dev/python-bootcamp/Lectures/04_IPythonNotebookIntroduction/mod.py in f(x)
      1 
      2 def f(x):
----> 3     return 1.0/(x-1)
      4 
      5 def g(y):

ZeroDivisionError: float division by zero

Plain exceptions

%xmode plain
mod.g(0)

Exception reporting mode: Plain

Traceback (most recent call last):

  File "<ipython-input-39-8932f4bf53fa>", line 2, in <module>
    mod.g(0)

  File "/Users/bussonniermatthias/dev/python-bootcamp/Lectures/04_IPythonNotebookIntroduction/mod.py", line 6, in g
    return f(y+1)

  File "/Users/bussonniermatthias/dev/python-bootcamp/Lectures/04_IPythonNotebookIntroduction/mod.py", line 3, in f
    return 1.0/(x-1)

ZeroDivisionError: float division by zero

Verbose exceptions

%xmode verbose
mod.g(0)

Exception reporting mode: Verbose

---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
<ipython-input-40-539f73e80e01> in <module>()
      1 get_ipython().magic('xmode verbose')
----> 2 mod.g(0)
        global mod.g = <function g at 0x106771840>

/Users/bussonniermatthias/dev/python-bootcamp/Lectures/04_IPythonNotebookIntroduction/mod.py in g(y=0)
      4 
      5 def g(y):
----> 6     return f(y+1)
        global f = <function f at 0x106754d08>
        y = 0

/Users/bussonniermatthias/dev/python-bootcamp/Lectures/04_IPythonNotebookIntroduction/mod.py in f(x=1)
      1 
      2 def f(x):
----> 3     return 1.0/(x-1)
        x = 1
      4 
      5 def g(y):

ZeroDivisionError: float division by zero

The default %xmode is "context", which shows additional context but not all local variables. Let's restore that one for the rest of our session.

%xmode context

Exception reporting mode: Context

Raw Input in the notebook

Jupyter notebook web application support raw_input(python2), or input (python3) which for example allow us to invoke the %debug magic in the notebook:

mod.g(0)

---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
<ipython-input-42-5e708f13c839> in <module>()
----> 1 mod.g(0)

/Users/bussonniermatthias/dev/python-bootcamp/Lectures/04_IPythonNotebookIntroduction/mod.py in g(y)
      4 
      5 def g(y):
----> 6     return f(y+1)

/Users/bussonniermatthias/dev/python-bootcamp/Lectures/04_IPythonNotebookIntroduction/mod.py in f(x)
      1 
      2 def f(x):
----> 3     return 1.0/(x-1)
      4 
      5 def g(y):

ZeroDivisionError: float division by zero

%debug

> /Users/bussonniermatthias/dev/python-bootcamp/Lectures/04_IPythonNotebookIntroduction/mod.py(3)f()
      1 
      2 def f(x):
----> 3     return 1.0/(x-1)
      4 
      5 def g(y):

ipdb> !x
1
ipdb> up
> /Users/bussonniermatthias/dev/python-bootcamp/Lectures/04_IPythonNotebookIntroduction/mod.py(6)g()
      2 def f(x):
      3     return 1.0/(x-1)
      4 
      5 def g(y):
----> 6     return f(y+1)

ipdb> exit

Don't foget to exit your debugging session. Raw input can of course be use to ask for user input:

enjoy = input('Are you enjoying this tutorial ?')
print('enjoy is :', enjoy)

Are you enjoying this tutorial ?A bit too fast
enjoy is : A bit too fast

Plotting in the notebook

This imports numpy as np and matplotlib's plotting routines as plt, plus setting lots of other stuff for you to work interactivel very easily:

%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt

Let's look at the results we collected earlier:

results[0].all_runs

[0.41974786203354597, 0.417751279019285, 0.4017370259971358]

[1]*3

[1, 1, 1]

fig, ax = plt.subplots()

for i,r in enumerate(results):
    ax.scatter([i+1]*len(r.all_runs),r.all_runs)


ax.set_title("An exponetial")

<matplotlib.text.Text at 0x1082e2978>

x = np.linspace(0, 2*np.pi, 300)
y = (1-np.exp(-x*0.5))

ax.set_ylim(0,1)
ax.set_xlim(0,6)

ax.plot(x, y)
ax.set_xlabel('More loops (Units)')
ax.set_ylabel('Measurements (s)')

<matplotlib.text.Text at 0x1082c0c18>

fig

The IPython kernel/client model

%connect_info

{
  "transport": "tcp",
  "shell_port": 57805,
  "stdin_port": 57807,
  "ip": "127.0.0.1",
  "kernel_name": "",
  "hb_port": 57809,
  "signature_scheme": "hmac-sha256",
  "iopub_port": 57806,
  "control_port": 57808,
  "key": "953de69d-9391-4cbb-b9d9-368dea98047d"
}

Paste the above JSON into a file, and connect with:
    $> ipython <app> --existing <file>
or, if you are local, you can connect with just:
    $> ipython <app> --existing /Users/bussonniermatthias/Library/Jupyter/runtime/kernel-1cbb81a5-42ea-443f-8db8-025d0d80d019.json 
or even just:
    $> ipython <app> --existing 
if this is the most recent IPython session you have started.

That's all folks!