In research and development (R&D), interactive computing environments are a frequently employed alternative for data exploration, algorithm development and prototyping. In the last twelve years, a popular scientific computing environment flourished around the Python programming language. Most of this environment is part of (or built over) a software stack named the SciPy Stack. Combined to the OpenCV’s Python interface, this environment becomes an alternative for current computer vision R&D. This tutorial introduces such an environment and shows how it can address different steps on computer vision research, from initial data exploration to parallel computing implementations. Several code examples are presented, addressing problems from simple image processing to inference by machine learning.

About these notebooks

These notebooks are part of the tutorial SciPy and OpenCV as an interactive computing environment for computer vision to be presented at SIBGRAPI 2014. The tutorial will be also available as a full paper in Revista de Informática Teórica e Aplicada (RITA).

Introduction

What are we looking for in a computing environment?

  • Data exploration and visualization
    • Images, video sequences, point clouds and feature vectors
  • Extensive list of tools
    • Image processing
    • Machine learning
    • Optimization
    • Statistics
    • Linear algebra
  • High-performance computing (HPC)
  • Promote reproducible research

Computer vision practitioners need a computing environment that lets them explore data like images, video sequences, point clouds and feature vectors. Such an environment should help on the development and testing of new models and algorithms, and on the deployment of the results, either as a final software module or a scientific/technical publication. It should also provides an extensive list of tools, routines for image processing, machine learning, statistical inference, linear algebra, convex optimization and graph algorithms, just to name a few. Problems involving large sets of images can require high-performance computing (HPC) such that the environment should provide practical ways to parallelize and distribute computation. An ideal environment should also combine documentation and computation in a single bundle, promoting results reproducibility, but preventing the research pipeline to become more cumbersome.

A Python-based environment

xkcd on Python

http://xkcd.com/353/ by Randall Munroe

  • SciPy, a scientific computing environment based on Python
    • Developed in the last 12 years
  • Based on the NumPy module for $n$-dimensional arrays
  • Not just software, but a community

In the last twelve years, a powerful scientific computing environment emerged from the Python programming community. This language is an attractive option for researchers: it is interpreted (a wanted property for interactive computing environments), dynamically typed, and presents a very concise and elegant syntax, resembling the pseudo-code found in computer science textbooks. But the tipping point for Python to become a major player in scientific computing was the advent of an efficient module for $n$-dimensional array representation and manipulation. The Numarray module was created by Greenfield et al. to address astronomical data analysis. In 2005, Numarray successor, NumPy, appeared and became the workhorse of the Python scientific computing. An active community composed by scientists and engineers flourished around Python and NumPy, represented today by the SciPy Stack and the SciPy Conferences.

Why is an interactive environment important to computer vision?

In computer vision, the practitioner is interested in inferring the world state from images, that act as observations. The statistical relation between the world state and the observed images is defined by models. A particular model is defined by parameters, chosen by learning algorithms. Finally, the world state is estimated by inference algorithms.

This vision on computer vision is properly presented by Prince in his book and translates the state-of-the-art of contemporary research in the field, which is deeply associated to machine learning nowadays.

  • Environment should enable tinkering with machine learning techniques
  • Visualization and exploration tools to address problems involving:
    • generalization,
    • overfitting,
    • dimensionality reducion,
    • optimization
  • The SciPy environment provides these capabilities

The development of these models and subsequent problems in learning and inference require a computing environment that allows proper tinkering with machine learning techniques. Visualization and exploration tools are necessary to address problems involving generalization, overfitting, dimensionality reduction and optimization. An interactive computing environment as IPython, enriched with tools from the SciPy Stack and the OpenCV library, can address these needs.

This tutorial introduces

  • interactive computing and HPC with the IPython shell;
  • OpenCV use under Python;
  • Matplotlib for visualization and 2D plotting;
  • the SciPy scientific library and
  • machine learning with the scikit-learn module
  • This tutorial provides a glimpse, not a broad and deep coverage
  • Examples should provide a starting point for the insterested user

The present tutorial provides a short overview on this Python-based computing environment. Considering the large set of tools available and the space constraints, this tutorial does not pretend to be a complete reference or a broad review. It provides just a glimpse of the environment's capabilities to the computer vision community, briefly presenting and discussing some problems and code examples. These examples can guide the user's first steps and the provided references help on the next ones.

Available material

IPython notebooks presenting the complete code for the examples are available on

http://nbviewer.ipython.org/github/thsant/scipy4cv

and on GitHub:

https://github.com/thsant/scipy4cv

Installation

Windows, Mac, and Linux

Consider Continuum Analytics’s Anaconda distribution:

http://continuum.io/downloads

Linux

  • Consider your system's package manager
  • For Debian/Ubuntu:
$ sudo apt-get install ipython ipython-qtconsole ipython-notebook

$ sudo apt-get install python-scipy python-matplotlib 

$ sudo apt-get install python-sklearn python-skimage

Using Python pip

$ pip install ipython jupyter scipy matplotlib scikit-learn scikit-image

OpenCV

As a general guideline, users should always consider the most recent installation instructions provided by the packages’ maintainers. To the date, Windows, Mac and Linux users could consider Continuum Analytics’s Anaconda distribution as one of the easiest ways to get all the environment presented in this tutorial. Linux users can also use the package manager of their systems to effortless download and install the components (as the apt-get system in Debian/Ubuntu distributions). The pip tool is also a practical way to get the packages and keep them updated.