Active Appearance Models (AAMs) are generative parametric models that describe the shape and appearance of a certain object class; e.g. the human face. In a typical application, these models are matched against input images to obtain the set of parameters that best describe a particular instance of the object being modelled.
The aim of this short notebook is to showcase how one can build and fit AAMs to images using Menpo
.
AAMs are typically build from a large collection of annotated images. In this notebook we will build AAMs from annotated images of the human face and, consequently, this is the object our AAMs will model.
The Breaking Bad image in Menpo
's data folder is a good example of this kind of annotated images:
%matplotlib inline
import menpo.io as mio
breakingbad = mio.import_builtin_asset('breakingbad.jpg')
breakingbad.crop_to_landmarks_proportion_inplace(0.5)
breakingbad.landmarks.view()
In this notebook, we will build AAMs using one of the most popular and widely used annotated facial databases, the Labeled Faces Parts in the Wild (LFPW) database. Both images and corresponding facial landmark annotations are publicly available and can be downloaded from the following link:
In order to continue with this notebook, the user is required to simply:
Note that the .zip
file containing the whole annotated database is of approximately 350MB.
path_to_lfpw = '/Users/joan/PhD/DataBases/'
The first step in building our AAM will be to import all the images and annotations of the training set of LFPW database. Luckily, Menpo
's io
module allows us to do exactly that using only a few lines of code:
import menpo.io as mio
training_images = []
# load landmarked images
for i in mio.import_images(path_to_lfpw + 'lfpw/trainset/*', verbose=True):
# crop image
i.crop_to_landmarks_proportion_inplace(0.1)
# convert it to greyscale if needed
if i.n_channels == 3:
i = i.as_greyscale(mode='luminosity')
# append it to the list
training_images.append(i)
- Loading 811 assets: [====================] 100%
The previous cell loads all the images of the LFPW together with their corresponding landmark annotations.
Note that here the images are cropped in order to save some valuable memory space and, for simplicity, also converted to greyscale.
In order to check if the data has been correctly imported we will define a simple visualization function using the widget package of IPython Notebook. All this function does is to use the interact
funtion to nicely display the a list of of annotated images.
import matplotlib.pyplot as plt
from IPython.html.widgets import interact
def browse_images(images, group=None, label='all'):
n = len(images)
def view_image(i):
images[i].landmarks[group][label].view()
plt.show()
interact(view_image, i=(0,n-1))
%matplotlib inline
browse_images(training_images)
We can visually check that the data has been correctly imported by simply playing with the slider bar on the previous cell.
Deformable models, in general, and AAMs, in particular, are a one of the core concept of Menpo
and all efforts have been put to facilitate their usage. In fact, given a list of training images, an AAM can be built using a single line of code.
from menpo.fitmultilevel.aam import AAMBuilder
# build AAM
aam = AAMBuilder(feature_type=None, normalization_diagonal=100, n_levels=1).build(training_images, verbose=True)
- Normalizing images size: Done - Computing feature space: Done - Building model - Done
As first class citizens of Menpo
, AAMs can be printed just like any other Menpo
object (e.g. Images
or PointClouds)
:
print aam
Active Appearance Model - 811 training images. - Warp using CachedPWA transform with 'scipy' interpolation. - No pyramid used: - No features extracted. 1 channel per image. - Reference frame of length 3708 (3708 x 1C, 75W x 74H x 1C) - 133 shape components (100.00% of variance) - 810 appearance components (100.00% of variance)
Printing an AAM is the easiest way to retrieve its specific characteristics. For example, printing the previous AAM tells us that it was build from 811 images among several other things.
AAMs also define an instance method that allows us to generate novel AAM instances by applying a set of particular weights to the components of their shape and apperance models:
%matplotlib inline
# using default parameters
aam.instance().view()
# varying shape parameters
aam.instance(shape_weights=[1.2, 0.5, -2.3]).view_new()
# varying appearance parameters
aam.instance(appearance_weights=[2.7, 3.5, 0.9]).view_new()
# varying both
aam.instance(shape_weights=[1.2, 0.5, -2.3], appearance_weights=[2.7, 3.5, 0.9]).view_new()
<menpo.visualize.viewmatplotlib.MatplotlibImageViewer2d at 0x1a6d0d10>
Furthermore, it is very simple to define powerfull visualization functions that allow us to explore AAMs using the previously introduced widget and the previous AAMs instance method:
def visualize_aam(aam, level=-1, bounds=(-3.0, 3.0), channels=None):
def view_aam(shape_pc1, shape_pc2, shape_pc3, tex_pc1, tex_pc2, tex_pc3):
instance = aam.instance(shape_weights=[shape_pc1, shape_pc2, shape_pc3],
appearance_weights=[tex_pc1, tex_pc2, tex_pc3],
level=level)
instance.landmarks.view(channels=channels)
plt.show()
interact(view_aam, shape_pc1=bounds, shape_pc2=bounds, shape_pc3=bounds, tex_pc1=bounds, tex_pc2=bounds, tex_pc3=bounds)
%matplotlib inline
visualize_aam(aam)
In Menpo
, AAMs can be fitted to images by creating Fitter
objects around them.
One of the most popular and well known family of algorithms for fitting AAMs is the one based around the original Lucas-Kanade
algorithm for Image Alignment. In order to fit our AAM using an algorithms of the previous family, Menpo
allows the user to define a LucasKanadeAAmFitter
object. Again, using a single line of code!!!
from menpo.fitmultilevel.aam import LucasKanadeAAMFitter
# define Lucas-Kanade based AAM fitter
fitter = LucasKanadeAAMFitter(aam, n_shape=0.9, n_appearance=0.9)
The previous cell has created a LucasKanadeAAMFitter
that will fit images using 90% of the variance present on the AAM's shape and appearance models and that will use the default LucasKanade
algorithm (i.e. the Alternating Inverse Compositional
algorithm) for fitting AAMs.
It is also important to know that these LucasKanadeAAMFitter
objects are also first class citizen in Menpo
and, consequently, can be printed:
print fitter
Active Appearance Model Fitter - Lucas-Kanade Alternating-IC - Transform is OrthoMDTransform and residual is SSD. - 811 training images. - No pyramid used: - No features extracted. 1 channel per image. - Reference frame of length 3708 (3708 x 1C, 75W x 74H x 1C) - 12 motion parameters - 30 appearance components (90.01% of original variance)
Fitting a LucasKanadeAAMFitter
to an image is as simple as calling its fit
method. Let's try it by fitting some images of the LFPW database test set!!!
import menpo.io as mio
# load test images
test_images = []
for i in mio.import_images(path_to_lfpw + 'lfpw/testset/*', max_images=5):
# crop image
i.crop_to_landmarks_proportion_inplace(0.5)
# convert it to grayscale if needed
if i.n_channels == 3:
i = i.as_greyscale(mode='luminosity')
# append it to the list
test_images.append(i)
Note that for the purpose of this simple fitting demonstration we will just fit the first 5 images of the LFPW test set.
fitting_results = []
# fit images
for j, i in enumerate(test_images):
# obtain groubnd truth (original) landmarks
gt_s = i.landmarks['PTS'].lms
# generate initialization landmarks
initial_s = fitter.perturb_shape(gt_s)
# fit image
fr = fitter.fit(i, initial_s, gt_shape=gt_s)
# append fitting result to list
fitting_results.append(fr)
# print image numebr
print 'Image: ', j
# print fitting error
print fr
Image: 0 Initial error: 0.1265 Final error: 0.0257 Image: 1 Initial error: 0.0806 Final error: 0.0725 Image: 2 Initial error: 0.0624 Final error: 0.0390 Image: 3 Initial error: 0.0881 Final error: 0.0590 Image: 4 Initial error: 0.0793 Final error: 0.0210
Menpo
's Fitter
objects save the result of their fittings using Menpo's FittingResult
objects. Being a Fitter
object itself, the LucasKanadeAAMFitter
is no exception to the rule and, consequently, the result obtained by executing the previous cell is a list of FittingResult
objects.
FittingResult
objects are core Menpo
objects that allow the user to print, visualize and analyse the results produced by Fitter
objects. Like all Menpo
's core objects FittingResult
are printable. Note that, they were printed inside of the previous fitting loop in order to display the final fitting error for each fitted image.
Apart from being printable, FittingResult
objects also allow the user to quickly visualize the initial shape from which the fitting algorithm started from and the final fitting result:
fr = fitting_results[3]
%matplotlib inline
# visualize initialization
fr.view_initialization(new_figure=True)
# visualize final fitting result
fr.view_final_fitting(new_figure=True)
<menpo.visualize.viewmatplotlib.MatplotlibLandmarkViewer2dImage at 0x22257790>