IPerl status and demo

author: Zaki Mughal

date: 2014-06-28

In [2]:
print "Hello! This IPython notebook file is generated with an early version of the IPerl language kernel.\n";
Out[2]:
Hello! This IPython notebook file is generated with an early version of the IPerl language kernel.
1

This document is meant to serve as a demonstration of how to run the IPerl kernel, what it can currently do, and what needs to be done to improve it.

Startup

You can get the source for the IPerl language kernel from GitHub by running

git clone https://github.com/zmughal/p5-Devel-IPerl.git

IPerl currently has some non-Perl dependencies that are needed for various features. These include the IPython and IPython notebook itself and some native code needed for communication (ZeroMQ) and content sniffing (libmagic). To install these dependencies on a Debian-based system run

sudo apt-get install libzmq3-dev ipython ipython-notebook libmagic-dev

You can then install the Perl dependencies using the cpanfile with cpanm in the top level of the source code directory

cpanm --installdeps .

Finally, to start up IPython notebook with the IPerl kernel, run

./bin/iperl notebook

Demo

Now let's see what the IPython notebook can do.

First, let's get the mandrill test image using the Data::TestImage module.

In [3]:
use Data::TestImage;
$mandrill_file = Data::TestImage->get_image('mandrill');
/home/zaki/perl5/perlbrew/perls/perl-5.16.3/lib/site_perl/5.16.3/auto/share/dist/Data-TestImage/USC/SIPI/misc/4.2.03.tiff

OK, that's just the path to a TIFF file. Now we can read it into a piddle. As you can see from the output, the number of dimensions in the PDL is 3. Note that the size of the last dimension is 3 which indicates that the image is an RGB image.

In [5]:
use PDL;
use DDP;
$mandrill = rim(~~ $mandrill_file); # need to stringify
p $mandrill;
PDL {
    Data     : too long to print
    Type     : unsigned char
    Shape    : [512 512 3]
    Nelem    : 786432
    Min      : 0
    Max      : 255
    Badflag  : No
    Has Bads : No
}

To display the image in IPython notebook, we need to have a PNG. We can write the piddle to a PNG file on disk and read it back in.

In [6]:
use Path::Class;
$temp_png_file = Path::Class::tempdir()->file( $mandrill_file->basename . '.png' );
$mandrill->wim( $temp_png_file );

IPerl->png( $mandrill_png = $temp_png_file->slurp );

Let's try something different. Here we use the Festival TTS engine to convert some text to a wave file of the generated speech. In order to play the sound in the browser, we need to convert the wave file into an MP3 file by using the lame encoder.

In [9]:
IPerl->audio_mpeg( $mp3 = qx{echo "Hello from IPerl" | text2wave | lame - -});
LAME 3.99.5 64bits (http://lame.sf.net)
Using polyphase lowpass filter, transition band:  5742 Hz -  5935 Hz
Encoding <stdin> to <stdout>
Encoding as 16 kHz single-ch MPEG-2 Layer III (10.7x)  24 kbps qval=3

We can also use PDL::Graphics::Gnuplot to plot 2 periods of a sine wave. This also needs to be written to a PNG file by passing the hardcopy option to the plot function.

In [11]:
use PDL::Graphics::Gnuplot;
use Math::Trig qw(pi);
         
$plot_png_file = Path::Class::tempdir()->file( 'plot.png' );
$x = zeroes(100)->xlinvals( -2 * pi, 2 * pi );
$w = gpwin();
$w->plot( $x, sin($x) , { xlabel => 'x', ylabel => 'sin(x)', xrange => [$x->minmax], hardcopy => $plot_png_file }  );
IPerl->png( $plot_png = $plot_png_file->slurp() );

The IPython notebook can also display LaTeX equations using the MathJax library:

\begin{equation} \int e^{x}\,d\!x = e^{x} + C \,. \end{equation}

Bugs

  • Handling output streams in an async way is currently not supported. For example:
    for (0..2) {
       say "test";
       sleep 2;
    }
    
    will take 6 seconds to run before it prints out "test" 3 times. I am assuming the best way to do this is to spawn a child process with redirected output handles and monitor its output. Perhaps by using IO::Pty. It is important to keep in mind how portable that may be.
  • Upon shutdown of the language kernel, ZeroMQ gives the following error:
        Bad file descriptor (epoll.cpp:67)
    I have a feeling this is because a part of the kernel is trying to read from a socket after the frontend has closed it.
  • The shell script for starting iperl (bin/iperl) needs to be converted to Perl for portability. I think that it should just be a wrapper around the ipython command except that it will also help create and load Perl profiles.
  • As always, the code needs more documentation, refactoring, logging, debugging tools and testing. The testing will require making a small frontend client. I think it needs more tests before it can go on CPAN and the only way to figure out what those tests should be like is by using the code.
  • The IPython messaging protocol uses ZeroMQ. This might make the kernel not as portable as it could be. There are several solutions to this.
    • Create a new wire protocol entirely using Perl and sockets. Frontends that support this protocol would need to be written.
    • Try to ensure ZeroMQ works everywhere. There exist the Alien::ZeroMQ and ZeroMQ::FFI distributions which could help avoid compilation on some platforms.
  • There are some execution semantics and message types in the messaging protocol that I haven't yet implemented/understood. I need to go through these and see if they are necessary.

Future features

  • Code completion in the code cells. Perhaps Padre and Farabi have implementations to look at.
  • A way to look up the POD documentation of a function/module.
  • Replace the small proof-of-concept content sniffing display hook with a pluggable system that has better support for MIME types and user configuration. It currently uses libmagic, so this needs to compile on all platforms. The Julia multimethod approach with MIME types and type pattern matching may be the way to go. See the multimedia I/O section of the base standard library. I am thinking that Type::Tiny could be useful here.
  • Work with Software Carpentry to create a Perl curriculum using IPerl?
  • I want to see bidirectional interactive JS working with IPerl. It would be neat to be able to be able to use Xtk for volume rendering and be able to get events back to Perl code and then call back to JS .
  • Support for D3.js using a matplotlib API is available through mpld3. It may be possible to provide a Perl API to the same JS.