ipythonPexpect example

Adam Lyon, Fermi National Accelerator Laboratory, lyon at fnal.gov, March 2013

ipythonPexpect is an IPython notebook extension that interfaces with the pexpect python module by Noah Spurrier. See https://pypi.python.org/pypi/pexpect/ and http://www.noah.org/wiki/Pexpect for detailed information on pexpect.

pexpect is a module that spawns a process that starts some application program (like a shell, R, CERN's Root, even Python). You can then feed it commands, capture the output, and wait for the next prompt. Most any command line driven application that provides a consistent prompt can be used. We'll do some examples here.

There is also a feature to lock the notebook such that all subsequent commands that you give, even without the magic, will go to the pexpect session. This is very convenient for long sessions, but may be surprising to other readers. So be sure to explain what you are doing carefully.

See https://cdcvs.fnal.gov/redmine/projects/ipython_ext/wiki/Wiki for more information including how to download this extension.

First, we load the notebook extension.

In [1]:
%reload_ext ipythonPexpect
%ipythonPexpect? for help

bash

There are some prewritten spawn scripts to start bash, R, CERN's ROOT, and python. Let's try a bash shell (note that prompt is changed to a known string).

In [2]:
%pexpect_spawn_bash
Opened connection to /usr/bin/env bash
PS1='bash> '
(pysci)mac-118277:examples lyon$ PS1='bash> '
bash> 

We can now send commands to this bash session with %%P.

In [3]:
%%P
date
bash> date
Tue Mar 12 23:40:01 CDT 2013
bash> 

pexpect sessions are persistent, so the state is maintained between commands (unlike ! which starts a new shell on each call)

In [4]:
!export FOO="hi"
In [5]:
!echo $FOO

In [6]:
%%P
export FOO="hi"
bash> export FOO="hi"
bash> 
In [7]:
%%P
echo $FOO
bash> echo $FOO
hi
bash> 

pexpect logs output to stdout so it appears in real time.

In [8]:
%%P
for i in $(seq 1 5); do sleep 2; date; done
bash> for i in $(seq 1 5); do sleep 2; date; done
Tue Mar 12 23:40:37 CDT 2013
Tue Mar 12 23:40:39 CDT 2013
Tue Mar 12 23:40:41 CDT 2013
Tue Mar 12 23:40:43 CDT 2013
Tue Mar 12 23:40:45 CDT 2013
bash> 

Note that blank lines and white space matter

In [9]:
%%P
   echo "hi"
    
bash>    echo "hi"
hi
bash> 
bash> 

When done, close the session

In [10]:
%pexpect_close
Closed connection to /usr/bin/env bash

python

Let's try a python session

In [11]:
%pexpect_spawn_python
Opened connection to /usr/bin/env python
Python 2.7.3 (default, Oct 30 2012, 01:05:24) 
[GCC 4.7.2] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 
In [12]:
%%P
print 'hi'
>>> print 'hi'
hi
>>> 
In [13]:
%%P
print 'hi'
print 'bye'
>>> print 'hi'
hi
>>> print 'bye'
bye
>>> 

For the command below, note the blank line at the end (for some reason, nbviewer doesn't show it, but it's there in the actual notebook)

In [14]:
%%P
for i in range(5):
  print i
>>> for i in range(5):
...   print i
... 
0
1
2
3
4
>>> 
In [15]:
%%P
print 'hi'+ \
  ' adam'
>>> print 'hi'+ \
...   ' adam'
hi adam
>>> 
In [16]:
%pexpect_close
Closed connection to /usr/bin/env python

R

Try R

In [17]:
%pexpect_spawn_R
Opened connection to R

R version 2.15.0 (2012-03-30)
Copyright (C) 2012 The R Foundation for Statistical Computing
ISBN 3-900051-07-0
Platform: x86_64-apple-darwin9.8.0/x86_64 (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> 
In [18]:
%%P
ls()
> ls()
character(0)
> 
In [19]:
%%P
a <- rnorm(20000)
head(a)
> a <- rnorm(20000)
> head(a)
[1] -0.98683074 -2.49823856 -0.68205272 -1.08784488  0.84639829 -0.01991325
> 

You can add an image if you know the output file name.

In [20]:
%%P -f rout.png
png("rout.png")
hist(a)
dev.off()
> png("rout.png")
> hist(a)
> dev.off()
null device 
          1 
> 
Out[20]:

You can return values to python (you may have to slice and dice the return string) with the -e (evaluate) option.

In [21]:
pya = %%P -e mean(a)
> mean(a)
[1] -0.005441818
> 
In [22]:
pya
Out[22]:
'[1] -0.005441818'
In [23]:
%pexpect_close
Closed connection to R

CERN ROOT

Note that ROOT is a special challenge because it does syntax highlighting and coloring at its command line. This leads to lots of ugly ANSI codes littering the output. The %pexpect_spawn_root magic writes a .rootrc file to the current directory that will turn Root to turn off this highlighting.

In [24]:
%pexpect_spawn_root
Opened connection to root -b
  *******************************************
  *                                         *
  *        W E L C O M E  to  R O O T       *
  *                                         *
  *   Version   5.34/03   27 October 2012   *
  *                                         *
  *  You are welcome to visit our Web site  *
  *          http://root.cern.ch            *
  *                                         *
  *******************************************

ROOT 5.34/03 (tags/v5-34-03@46856, Oct 27 2012, 23:19:27 on macosx64)

CINT/ROOT C/C++ Interpreter version 5.18.00, July 2, 2010
Type ? for help. Commands must be C++ statements.
Enclose multiple statements between { }.
root [0] 

We're going to try the phase space tutorial

In [27]:
%%P
gSystem.Load("libPhysics")
root [1] gSystem.Load("libPhysics")
(int)1
root [2] 

I'm getting tired of entering %%P. I'm going to lock the notebook in Root mode. Make sure you know what you are doing here. When I'm done, I have to %pexpect_unlock in order for the notebook to process IPython again.

In [28]:
%pexpect_lock
WARNING: All future cell execution will be processed through pexpect!
To return to IPython, issue %pexpect_unlock

Continuing onward...

In [29]:
TLorentzVector target(0.0, 0.0, 0.0, 0.938)
TLorentzVector beam(0.0, 0.0, 0.65, 0.65)
TLorentzVector W = beam + target
root [2] TLorentzVector target(0.0, 0.0, 0.0, 0.938)
root [3] TLorentzVector beam(0.0, 0.0, 0.65, 0.65)
root [4] TLorentzVector W = beam + target
root [5] 
In [30]:
Double_t masses[3] = { 0.938, 0.139, 0.139 }
root [5] Double_t masses[3] = { 0.938, 0.139, 0.139 }
root [6] 
In [31]:
masses
root [6] masses
(Double_t*)0x7feaa1a77d40
root [7] 
In [32]:
TGenPhaseSpace event
root [7] TGenPhaseSpace event
root [8] 
In [33]:
event.SetDecay(W, 3, masses)
root [8] event.SetDecay(W, 3, masses)
(Bool_t)1
root [9] 
In [34]:
 TH2F *h2 = new TH2F("h2","h2", 50,1.1,1.8, 50,1.1,1.8);
root [9]  TH2F *h2 = new TH2F("h2","h2", 50,1.1,1.8, 50,1.1,1.8);
root [10] 

For some reason, if you don't put blocks of code within braces, you get syntax coloring again.

In [35]:
{
  for (Int_t n=0;n<100000;n++) {
      Double_t weight = event.Generate();

      TLorentzVector *pProton = event.GetDecay(0);

      TLorentzVector *pPip    = event.GetDecay(1);
      TLorentzVector *pPim    = event.GetDecay(2);

      TLorentzVector pPPip = *pProton + *pPip;
      TLorentzVector pPPim = *pProton + *pPim;

      h2->Fill(pPPip.M2() ,pPPim.M2() ,weight);
   }
}
root [10] {
end with '}', '@':abort >   for (Int_t n=0;n<100000;n++) {
end with '}', '@':abort >       Double_t weight = event.Generate();
end with '}', '@':abort > 
end with '}', '@':abort >       TLorentzVector *pProton = event.GetDecay(0);
end with '}', '@':abort > 
end with '}', '@':abort >       TLorentzVector *pPip    = event.GetDecay(1);
end with '}', '@':abort >       TLorentzVector *pPim    = event.GetDecay(2);
end with '}', '@':abort > 
end with '}', '@':abort >       TLorentzVector pPPip = *pProton + *pPip;
end with '}', '@':abort >       TLorentzVector pPPim = *pProton + *pPim;
end with '}', '@':abort > 
end with '}', '@':abort >       h2->Fill(pPPip.M2() ,pPPim.M2() ,weight);
end with '}', '@':abort >    }
end with '}', '@':abort > }
root [11] 
In [36]:
h2
root [11] h2
(class TH2F*)0x7feaa1b2bc30
root [12] 
In [37]:
h2->Draw()
root [12] h2->Draw()
Info in <TCanvas::MakeDefCanvas>:  created default TCanvas with name c1
root [13] 

Event though I'm in "locked" mode, I can still issue %%P if I need to give it options.

In [39]:
%%P -f root.png
c1.SaveAs("root.png")
root [14] c1.SaveAs("root.png")
Info in <TCanvas::Print>: png file root.png has been created
root [15] 
Out[39]:
In [40]:
%pexpect_unlock
Notebook will use IPython
In [41]:
%pexpect_close
Closed connection to root -b

Your own application

Use the %pexpect_spawn command and be sure to ask the system for help for the options.

In [42]:
%pexpect_spawn?

Let's try to connect to ruby. Note that irb is a little strange - when it first comes up it shows the prompt with no blank lines in front. So make a litle initialization to fix that.

In [47]:
%pexpect_spawn -p '\r\n>> ' -c '\r\n\?> ' -e irb -i "2+2"
Opened connection to irb
2+2>> 
2+2
=> 4
>> 
In [48]:
%%P
4
>> 4
=> 4
>> 
In [49]:
%%P
puts("hi")
>> puts("hi")
hi
=> nil
>> 
In [50]:
%pexpect_close
Closed connection to irb