Using Fortran and C code with Python

J.R. Johansson (robert@riken.jp) http://dml.riken.jp/~rob/

The latest version of this IPython notebook lecture is available at http://github.com/jrjohansson/scientific-python-lectures.

The other notebooks in this lecture series are indexed at http://jrjohansson.github.com.

In [1]:
%pylab inline
from IPython.display import Image

Welcome to pylab, a matplotlib-based Python environment [backend: module://IPython.zmq.pylab.backend_inline].
For more information, type 'help(pylab)'.

The advantage of Python is that it is flexible and easy to program. The time it takes to setup a new calulation is therefore short. But for certain types of calculations Python (and any other interpreted language) can be very slow. It is particularly iterations over large arrays that is difficult to do efficiently.

Such calculations may be implemented in a compiled language such as C or Fortran. In Python it is relatively easy to call out to libraries with compiled C or Fortran code. In this lecture we will look at how to do that.

But before we go ahead and work on optimizing anything, it is always worthwhile to ask....

In [2]:
Image(filename='images/optimizing-what.png')
Out[2]:

Fortran

F2PY

F2PY is a program that (almost) automatically wraps fortran code for use in Python: By using the f2py program we can compile fortran code into a module that we can import in a Python program.

F2PY is a part of NumPy, but you will also need to have a fortran compiler to run the examples below.

Example 0: scalar input, no output

In [3]:
%%file hellofortran.f
C File  hellofortran.f
        subroutine hellofortran (n)
        integer n
       
        do 100 i=0, n
            print *, "Fortran says hello"
100     continue
        end
Overwriting hellofortran.f

Generate a python module using f2py:

In [4]:
!f2py -c -m hellofortran hellofortran.f
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_src
build_src
building extension "hellofortran" sources
f2py options: []
f2py:> /tmp/tmp7TbvkO/src.linux-x86_64-2.7/hellofortranmodule.c
creating /tmp/tmp7TbvkO
creating /tmp/tmp7TbvkO/src.linux-x86_64-2.7
Reading fortran codes...
	Reading file 'hellofortran.f' (format:fix,strict)
Post-processing...
	Block: hellofortran
			Block: hellofortran
Post-processing (stage 2)...
Building modules...
	Building module "hellofortran"...
		Constructing wrapper function "hellofortran"...
		  hellofortran(n)
	Wrote C/API module "hellofortran" to file "/tmp/tmp7TbvkO/src.linux-x86_64-2.7/hellofortranmodule.c"
  adding '/tmp/tmp7TbvkO/src.linux-x86_64-2.7/fortranobject.c' to sources.
  adding '/tmp/tmp7TbvkO/src.linux-x86_64-2.7' to include_dirs.
copying /usr/lib/python2.7/dist-packages/numpy/f2py/src/fortranobject.c -> /tmp/tmp7TbvkO/src.linux-x86_64-2.7
copying /usr/lib/python2.7/dist-packages/numpy/f2py/src/fortranobject.h -> /tmp/tmp7TbvkO/src.linux-x86_64-2.7
build_src: building npy-pkg config files
running build_ext
customize UnixCCompiler
customize UnixCCompiler using build_ext
customize Gnu95FCompiler
Found executable /usr/bin/gfortran
customize Gnu95FCompiler
customize Gnu95FCompiler using build_ext
building 'hellofortran' extension
compiling C sources
C compiler: x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC

creating /tmp/tmp7TbvkO/tmp
creating /tmp/tmp7TbvkO/tmp/tmp7TbvkO
creating /tmp/tmp7TbvkO/tmp/tmp7TbvkO/src.linux-x86_64-2.7
compile options: '-I/tmp/tmp7TbvkO/src.linux-x86_64-2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c'
x86_64-linux-gnu-gcc: /tmp/tmp7TbvkO/src.linux-x86_64-2.7/fortranobject.c
In file included from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1728:0,
                 from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarrayobject.h:17,
                 from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/arrayobject.h:15,
                 from /tmp/tmp7TbvkO/src.linux-x86_64-2.7/fortranobject.h:13,
                 from /tmp/tmp7TbvkO/src.linux-x86_64-2.7/fortranobject.c:2:
/usr/lib/python2.7/dist-packages/numpy/core/include/numpy/npy_deprecated_api.h:11:2: warning: #warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
x86_64-linux-gnu-gcc: /tmp/tmp7TbvkO/src.linux-x86_64-2.7/hellofortranmodule.c
In file included from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1728:0,
                 from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarrayobject.h:17,
                 from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/arrayobject.h:15,
                 from /tmp/tmp7TbvkO/src.linux-x86_64-2.7/fortranobject.h:13,
                 from /tmp/tmp7TbvkO/src.linux-x86_64-2.7/hellofortranmodule.c:17:
/usr/lib/python2.7/dist-packages/numpy/core/include/numpy/npy_deprecated_api.h:11:2: warning: #warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
compiling Fortran sources
Fortran f77 compiler: /usr/bin/gfortran -Wall -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran f90 compiler: /usr/bin/gfortran -Wall -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran fix compiler: /usr/bin/gfortran -Wall -ffixed-form -fno-second-underscore -Wall -fno-second-underscore -fPIC -O3 -funroll-loops
compile options: '-I/tmp/tmp7TbvkO/src.linux-x86_64-2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c'
gfortran:f77: hellofortran.f
/usr/bin/gfortran -Wall -Wall -shared /tmp/tmp7TbvkO/tmp/tmp7TbvkO/src.linux-x86_64-2.7/hellofortranmodule.o /tmp/tmp7TbvkO/tmp/tmp7TbvkO/src.linux-x86_64-2.7/fortranobject.o /tmp/tmp7TbvkO/hellofortran.o -lgfortran -o ./hellofortran.so
running scons
Removing build directory /tmp/tmp7TbvkO

Example of a python script that use the module:

In [5]:
%%file hello.py
import hellofortran

hellofortran.hellofortran(5)
Overwriting hello.py

In [6]:
# run the script
!python hello.py
 Fortran says hello
 Fortran says hello
 Fortran says hello
 Fortran says hello
 Fortran says hello
 Fortran says hello

Example 1: vector input and scalar output

In [7]:
%%file dprod.f

       subroutine dprod(x, y, n)
    
       double precision x(n), y
       y = 1.0
    
       do 100 i=1, n
           y = y * x(i)
100    continue
       end
Overwriting dprod.f

In [8]:
!rm -f dprod.pyf
!f2py -m dprod -h dprod.pyf dprod.f
Reading fortran codes...
	Reading file 'dprod.f' (format:fix,strict)
Post-processing...
	Block: dprod
{}
In: :dprod:dprod.f:dprod
vars2fortran: No typespec for argument "n".
			Block: dprod
Post-processing (stage 2)...
Saving signatures to file "./dprod.pyf"

The f2py program generated a module declaration file called dsum.pyf. Let's look what's in it:

In [9]:
!cat dprod.pyf
!    -*- f90 -*-
! Note: the context of this file is case sensitive.

python module dprod ! in 
    interface  ! in :dprod
        subroutine dprod(x,y,n) ! in :dprod:dprod.f
            double precision dimension(n) :: x
            double precision :: y
            integer, optional,check(len(x)>=n),depend(x) :: n=len(x)
        end subroutine dprod
    end interface 
end python module dprod

! This file was auto-generated with f2py (version:2).
! See http://cens.ioc.ee/projects/f2py2e/

The module does not know what Fortran subroutine arguments is input and output, so we need to manually edit the module declaration files and mark output variables with intent(out) and input variable with intent(in):

In [10]:
%%file dprod.pyf
python module dprod ! in 
    interface  ! in :dprod
        subroutine dprod(x,y,n) ! in :dprod:dprod.f
            double precision dimension(n), intent(in) :: x
            double precision, intent(out) :: y
            integer, optional,check(len(x)>=n),depend(x),intent(in) :: n=len(x)
        end subroutine dprod
    end interface 
end python module dprod
Overwriting dprod.pyf

Compile the fortran code into a module that can be included in python:

In [11]:
!f2py -c dprod.pyf dprod.f
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_src
build_src
building extension "dprod" sources
creating /tmp/tmp8NeWyx
creating /tmp/tmp8NeWyx/src.linux-x86_64-2.7
f2py options: []
f2py: dprod.pyf
Reading fortran codes...
	Reading file 'dprod.pyf' (format:free)
Post-processing...
	Block: dprod
			Block: dprod
Post-processing (stage 2)...
Building modules...
	Building module "dprod"...
		Constructing wrapper function "dprod"...
		  y = dprod(x,[n])
	Wrote C/API module "dprod" to file "/tmp/tmp8NeWyx/src.linux-x86_64-2.7/dprodmodule.c"
  adding '/tmp/tmp8NeWyx/src.linux-x86_64-2.7/fortranobject.c' to sources.
  adding '/tmp/tmp8NeWyx/src.linux-x86_64-2.7' to include_dirs.
copying /usr/lib/python2.7/dist-packages/numpy/f2py/src/fortranobject.c -> /tmp/tmp8NeWyx/src.linux-x86_64-2.7
copying /usr/lib/python2.7/dist-packages/numpy/f2py/src/fortranobject.h -> /tmp/tmp8NeWyx/src.linux-x86_64-2.7
build_src: building npy-pkg config files
running build_ext
customize UnixCCompiler
customize UnixCCompiler using build_ext
customize Gnu95FCompiler
Found executable /usr/bin/gfortran
customize Gnu95FCompiler
customize Gnu95FCompiler using build_ext
building 'dprod' extension
compiling C sources
C compiler: x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC

creating /tmp/tmp8NeWyx/tmp
creating /tmp/tmp8NeWyx/tmp/tmp8NeWyx
creating /tmp/tmp8NeWyx/tmp/tmp8NeWyx/src.linux-x86_64-2.7
compile options: '-I/tmp/tmp8NeWyx/src.linux-x86_64-2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c'
x86_64-linux-gnu-gcc: /tmp/tmp8NeWyx/src.linux-x86_64-2.7/dprodmodule.c
In file included from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1728:0,
                 from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarrayobject.h:17,
                 from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/arrayobject.h:15,
                 from /tmp/tmp8NeWyx/src.linux-x86_64-2.7/fortranobject.h:13,
                 from /tmp/tmp8NeWyx/src.linux-x86_64-2.7/dprodmodule.c:18:
/usr/lib/python2.7/dist-packages/numpy/core/include/numpy/npy_deprecated_api.h:11:2: warning: #warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
/tmp/tmp8NeWyx/src.linux-x86_64-2.7/dprodmodule.c:111:12: warning: ‘f2py_size’ defined but not used [-Wunused-function]
x86_64-linux-gnu-gcc: /tmp/tmp8NeWyx/src.linux-x86_64-2.7/fortranobject.c
In file included from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1728:0,
                 from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarrayobject.h:17,
                 from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/arrayobject.h:15,
                 from /tmp/tmp8NeWyx/src.linux-x86_64-2.7/fortranobject.h:13,
                 from /tmp/tmp8NeWyx/src.linux-x86_64-2.7/fortranobject.c:2:
/usr/lib/python2.7/dist-packages/numpy/core/include/numpy/npy_deprecated_api.h:11:2: warning: #warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
compiling Fortran sources
Fortran f77 compiler: /usr/bin/gfortran -Wall -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran f90 compiler: /usr/bin/gfortran -Wall -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran fix compiler: /usr/bin/gfortran -Wall -ffixed-form -fno-second-underscore -Wall -fno-second-underscore -fPIC -O3 -funroll-loops
compile options: '-I/tmp/tmp8NeWyx/src.linux-x86_64-2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c'
gfortran:f77: dprod.f
/usr/bin/gfortran -Wall -Wall -shared /tmp/tmp8NeWyx/tmp/tmp8NeWyx/src.linux-x86_64-2.7/dprodmodule.o /tmp/tmp8NeWyx/tmp/tmp8NeWyx/src.linux-x86_64-2.7/fortranobject.o /tmp/tmp8NeWyx/dprod.o -lgfortran -o ./dprod.so
running scons
Removing build directory /tmp/tmp8NeWyx

Using the module from Python

In [12]:
import dprod
In [13]:
help(dprod)
Help on module dprod:

NAME
    dprod

FILE
    /home/rob/Desktop/scientific-python-lectures/dprod.so

DESCRIPTION
    This module 'dprod' is auto-generated with f2py (version:2).
    Functions:
      y = dprod(x,n=len(x))
    .

DATA
    __version__ = '$Revision: $'
    dprod = <fortran object>

VERSION




In [14]:
dprod.dprod(arange(1,50))
Out[14]:
6.082818640342675e+62
In [15]:
# compare to numpy
prod(arange(1.0,50.0))
Out[15]:
6.0828186403426752e+62
In [16]:
dprod.dprod(arange(1,10), 5) # only the 5 first elements
Out[16]:
120.0

Compare performance:

In [17]:
xvec = rand(500)
In [18]:
timeit dprod.dprod(xvec)
1000000 loops, best of 3: 896 ns per loop

In [19]:
timeit xvec.prod()
100000 loops, best of 3: 4.44 us per loop

Example 2: cummulative sum, vector input and vector output

The cummulative sum function for an array of data is a good example of a loop intense algorithm: Loop through a vector and store the cummulative sum in another vector.

In [20]:
# simple python algorithm: example of a SLOW implementation
# Why? Because the loop is implemented in python.
def py_dcumsum(a):
    b = empty_like(a)
    b[0] = a[0]
    for n in range(1,len(a)):
        b[n] = b[n-1]+a[n]
    return b

Fortran subroutine for the same thing: here we have added the intent(in) and intent(out) as comment lines in the original fortran code, so we do not need to manually edit the fortran module declaration file generated by f2py.

In [21]:
%%file dcumsum.f
c File dcumsum.f
       subroutine dcumsum(a, b, n)
       double precision a(n)
       double precision b(n)
       integer n
cf2py  intent(in) :: a
cf2py  intent(out) :: b
cf2py  intent(hide) :: n

       b(1) = a(1)
       do 100 i=2, n
           b(i) = b(i-1) + a(i)
100    continue
       end
Overwriting dcumsum.f

We can directly compile the fortran code to a python module:

In [22]:
!f2py -c dcumsum.f -m dcumsum
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_src
build_src
building extension "dcumsum" sources
f2py options: []
f2py:> /tmp/tmpy66yrA/src.linux-x86_64-2.7/dcumsummodule.c
creating /tmp/tmpy66yrA
creating /tmp/tmpy66yrA/src.linux-x86_64-2.7
Reading fortran codes...
	Reading file 'dcumsum.f' (format:fix,strict)
Post-processing...
	Block: dcumsum
			Block: dcumsum
Post-processing (stage 2)...
Building modules...
	Building module "dcumsum"...
		Constructing wrapper function "dcumsum"...
		  b = dcumsum(a)
	Wrote C/API module "dcumsum" to file "/tmp/tmpy66yrA/src.linux-x86_64-2.7/dcumsummodule.c"
  adding '/tmp/tmpy66yrA/src.linux-x86_64-2.7/fortranobject.c' to sources.
  adding '/tmp/tmpy66yrA/src.linux-x86_64-2.7' to include_dirs.
copying /usr/lib/python2.7/dist-packages/numpy/f2py/src/fortranobject.c -> /tmp/tmpy66yrA/src.linux-x86_64-2.7
copying /usr/lib/python2.7/dist-packages/numpy/f2py/src/fortranobject.h -> /tmp/tmpy66yrA/src.linux-x86_64-2.7
build_src: building npy-pkg config files
running build_ext
customize UnixCCompiler
customize UnixCCompiler using build_ext
customize Gnu95FCompiler
Found executable /usr/bin/gfortran
customize Gnu95FCompiler
customize Gnu95FCompiler using build_ext
building 'dcumsum' extension
compiling C sources
C compiler: x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC

creating /tmp/tmpy66yrA/tmp
creating /tmp/tmpy66yrA/tmp/tmpy66yrA
creating /tmp/tmpy66yrA/tmp/tmpy66yrA/src.linux-x86_64-2.7
compile options: '-I/tmp/tmpy66yrA/src.linux-x86_64-2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c'
x86_64-linux-gnu-gcc: /tmp/tmpy66yrA/src.linux-x86_64-2.7/fortranobject.c
In file included from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1728:0,
                 from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarrayobject.h:17,
                 from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/arrayobject.h:15,
                 from /tmp/tmpy66yrA/src.linux-x86_64-2.7/fortranobject.h:13,
                 from /tmp/tmpy66yrA/src.linux-x86_64-2.7/fortranobject.c:2:
/usr/lib/python2.7/dist-packages/numpy/core/include/numpy/npy_deprecated_api.h:11:2: warning: #warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
x86_64-linux-gnu-gcc: /tmp/tmpy66yrA/src.linux-x86_64-2.7/dcumsummodule.c
In file included from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1728:0,
                 from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/ndarrayobject.h:17,
                 from /usr/lib/python2.7/dist-packages/numpy/core/include/numpy/arrayobject.h:15,
                 from /tmp/tmpy66yrA/src.linux-x86_64-2.7/fortranobject.h:13,
                 from /tmp/tmpy66yrA/src.linux-x86_64-2.7/dcumsummodule.c:18:
/usr/lib/python2.7/dist-packages/numpy/core/include/numpy/npy_deprecated_api.h:11:2: warning: #warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
/tmp/tmpy66yrA/src.linux-x86_64-2.7/dcumsummodule.c:111:12: warning: ‘f2py_size’ defined but not used [-Wunused-function]
compiling Fortran sources
Fortran f77 compiler: /usr/bin/gfortran -Wall -ffixed-form -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran f90 compiler: /usr/bin/gfortran -Wall -fno-second-underscore -fPIC -O3 -funroll-loops
Fortran fix compiler: /usr/bin/gfortran -Wall -ffixed-form -fno-second-underscore -Wall -fno-second-underscore -fPIC -O3 -funroll-loops
compile options: '-I/tmp/tmpy66yrA/src.linux-x86_64-2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -c'
gfortran:f77: dcumsum.f
/usr/bin/gfortran -Wall -Wall -shared /tmp/tmpy66yrA/tmp/tmpy66yrA/src.linux-x86_64-2.7/dcumsummodule.o /tmp/tmpy66yrA/tmp/tmpy66yrA/src.linux-x86_64-2.7/fortranobject.o /tmp/tmpy66yrA/dcumsum.o -lgfortran -o ./dcumsum.so
running scons
Removing build directory /tmp/tmpy66yrA

In [23]:
import dcumsum
In [24]:
a = array([1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0])
In [25]:
py_dcumsum(a)
Out[25]:
array([  1.,   3.,   6.,  10.,  15.,  21.,  28.,  36.])
In [26]:
dcumsum.dcumsum(a)
Out[26]:
array([  1.,   3.,   6.,  10.,  15.,  21.,  28.,  36.])
In [27]:
cumsum(a)
Out[27]:
array([  1.,   3.,   6.,  10.,  15.,  21.,  28.,  36.])

Benchmark the different implementations:

In [28]:
a = rand(10000)
In [29]:
timeit py_dcumsum(a)
100 loops, best of 3: 5.16 ms per loop

In [30]:
timeit dcumsum.dcumsum(a)
100000 loops, best of 3: 11.4 us per loop

In [31]:
timeit a.cumsum()
10000 loops, best of 3: 27.1 us per loop

C

ctypes

ctypes is a Python library for calling out to C code. It is not as automatic as f2py, and we manually need to load the library and set properties such as the functions return and argument types. On the otherhand we do not need to touch the C code at all.

In [32]:
%%file functions.c

#include <stdio.h>

void hello(int n);

double dprod(double *x, int n);

void dcumsum(double *a, double *b, int n);

void
hello(int n)
{
    int i;
    
    for (i = 0; i < n; i++)
    {
        printf("C says hello\n");
    }
}


double 
dprod(double *x, int n)
{
    int i;
    double y = 1.0;
    
    for (i = 0; i < n; i++)
    {
        y *= x[i];
    }

    return y;
}

void
dcumsum(double *a, double *b, int n)
{
    int i;
    
    b[0] = a[0];
    for (i = 1; i < n; i++)
    {
        b[i] = a[i] + b[i-1];
    }
}
Overwriting functions.c

Compile the C file into a shared library:

In [33]:
!gcc -c -Wall -O2 -Wall -ansi -pedantic -fPIC -o functions.o functions.c
!gcc -o libfunctions.so -shared functions.o

The result is a compiled shared library libfunctions.so:

In [34]:
!file libfunctions.so
libfunctions.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0xe88d287644685321aaf94ccb28f5ec546b2cf6a0, not stripped

Now we need to write wrapper functions to access the C library: To load the library we use the ctypes package, which included in the Python standard library (with extensions from numpy for passing arrays to C). Then we manually set the types of the argument and return values (no automatic code inspection here!).

In [35]:
%%file functions.py

import numpy
import ctypes

_libfunctions = numpy.ctypeslib.load_library('libfunctions', '.')

_libfunctions.hello.argtypes = [ctypes.c_int]
_libfunctions.hello.restype  =  ctypes.c_void_p

_libfunctions.dprod.argtypes = [numpy.ctypeslib.ndpointer(dtype=numpy.float), ctypes.c_int]
_libfunctions.dprod.restype  = ctypes.c_double

_libfunctions.dcumsum.argtypes = [numpy.ctypeslib.ndpointer(dtype=numpy.float), numpy.ctypeslib.ndpointer(dtype=numpy.float), ctypes.c_int]
_libfunctions.dcumsum.restype  = ctypes.c_void_p

def hello(n):
    return _libfunctions.hello(int(n))

def dprod(x, n=None):
    if n is None:
        n = len(x)
    x = numpy.asarray(x, dtype=numpy.float)
    return _libfunctions.dprod(x, int(n))

def dcumsum(a, n):
    a = numpy.asarray(a, dtype=numpy.float)
    b = numpy.empty(len(a), dtype=numpy.float)
    _libfunctions.dcumsum(a, b, int(n))
    return b
Overwriting functions.py

In [36]:
%%file run_hello_c.py

import functions

functions.hello(3)
Overwriting run_hello_c.py

In [37]:
!python run_hello_c.py
C says hello
C says hello
C says hello

In [38]:
import functions

Product function:

In [39]:
functions.dprod([1,2,3,4,5]) 
Out[39]:
120.0

Cummulative sum:

In [40]:
a = rand(100000)
In [41]:
res_c = functions.dcumsum(a, len(a)) 
In [42]:
res_fortran = dcumsum.dcumsum(a)
In [43]:
res_c - res_fortran
Out[43]:
array([ 0.,  0.,  0., ...,  0.,  0.,  0.])

Simple benchmark

In [44]:
timeit functions.dcumsum(a, len(a))
1000 loops, best of 3: 290 us per loop

In [45]:
timeit dcumsum.dcumsum(a)
10000 loops, best of 3: 118 us per loop

In [46]:
timeit a.cumsum()
1000 loops, best of 3: 262 us per loop

Cython

A hybrid between python and C that can be compiled: Basically Python code with type declarations.

In [47]:
%%file cy_dcumsum.pyx

cimport numpy

def dcumsum(numpy.ndarray[numpy.float64_t, ndim=1] a, numpy.ndarray[numpy.float64_t, ndim=1] b):
    cdef int i, n = len(a)
    b[0] = a[0]
    for i from 1 <= i < n:
        b[i] = b[i-1] + a[i]
    return b
Overwriting cy_dcumsum.pyx

A build file for generating C code and compiling it into a Python module.

In [48]:
%%file setup.py

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

setup(
    cmdclass = {'build_ext': build_ext},
    ext_modules = [Extension("cy_dcumsum", ["cy_dcumsum.pyx"])]
)
Overwriting setup.py

In [49]:
!python setup.py build_ext --inplace
running build_ext
cythoning cy_dcumsum.pyx to cy_dcumsum.c
building 'cy_dcumsum' extension
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c cy_dcumsum.c -o build/temp.linux-x86_64-2.7/cy_dcumsum.o
In file included from /usr/include/python2.7/numpy/ndarraytypes.h:1728:0,
                 from /usr/include/python2.7/numpy/ndarrayobject.h:17,
                 from /usr/include/python2.7/numpy/arrayobject.h:15,
                 from cy_dcumsum.c:256:
/usr/include/python2.7/numpy/npy_deprecated_api.h:11:2: warning: #warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
In file included from /usr/include/python2.7/numpy/ndarrayobject.h:26:0,
                 from /usr/include/python2.7/numpy/arrayobject.h:15,
                 from cy_dcumsum.c:256:
/usr/include/python2.7/numpy/__multiarray_api.h:1594:1: warning: ‘_import_array’ defined but not used [-Wunused-function]
In file included from /usr/include/python2.7/numpy/ufuncobject.h:311:0,
                 from cy_dcumsum.c:257:
/usr/include/python2.7/numpy/__ufunc_api.h:236:1: warning: ‘_import_umath’ defined but not used [-Wunused-function]
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/cy_dcumsum.o -o /home/rob/Desktop/scientific-python-lectures/cy_dcumsum.so

In [50]:
import cy_dcumsum
In [51]:
a = array([1,2,3,4], dtype=float)
b = empty_like(a)
cy_dcumsum.dcumsum(a,b)
b
Out[51]:
array([  1.,   3.,   6.,  10.])
In [52]:
a = array([1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0])
In [53]:
b = empty_like(a)
cy_dcumsum.dcumsum(a, b)
b
Out[53]:
array([  1.,   3.,   6.,  10.,  15.,  21.,  28.,  36.])
In [54]:
py_dcumsum(a)
Out[54]:
array([  1.,   3.,   6.,  10.,  15.,  21.,  28.,  36.])
In [55]:
a = rand(100000)
b = empty_like(a)
In [56]:
timeit py_dcumsum(a)
10 loops, best of 3: 48.2 ms per loop

In [57]:
timeit cy_dcumsum.dcumsum(a,b)
1000 loops, best of 3: 263 us per loop

Cython in the IPython notebook

When working with the IPython (especially in the notebook), there is a more convenient way of compiling and loading Cython code. Using the %%cython IPython magic (command to IPython), we can simply type the Cython code in a code cell and let IPython take care of the conversion to C code, compilation and loading of the function. To be able to use the %%cython magic, we first need to load the extension cythonmagic:

In [58]:
%load_ext cythonmagic
In [59]:
%%cython

cimport numpy

def cy_dcumsum2(numpy.ndarray[numpy.float64_t, ndim=1] a, numpy.ndarray[numpy.float64_t, ndim=1] b):
    cdef int i, n = len(a)
    b[0] = a[0]
    for i from 1 <= i < n:
        b[i] = b[i-1] + a[i]
    return b
In [60]:
timeit cy_dcumsum2(a,b)
1000 loops, best of 3: 265 us per loop

Back to top