Notebook

Introduction to Astronomical Image Processing Pipelines : Automating the Process of Discovery¶

This notebook outlines in detail how to perform photometry on astronomical images. We will assume that the images have already been calibrated, in the sense that the instrument signatures have been removed. The goal is to measure the brightness of each source within the image, and demonstrate how to undertake optimal point-source photometry by building and using a point spread function for the image.

We will demonstrate two sets of packages. The first part takes a detailed look at the internals of the process using the LSST Data Management software stack (https://confluence.lsstcorp.org/display/LSWUG/Getting+Started+with+the+LSST+Software+Stack). The second approach (on Monday) outlines an equivalent analysis using the combination of SExtractor (http://www.astromatic.net/software/sextractor) and PSFEx (http://www.astromatic.net/software/psfex). Oh, and you'll also need ds9 (http://ds9.si.edu/site/Download.html).

To "set up" the LSST code, and then run this notebook, run the commands below:

cd /location/to/lsst/code source loadLSST.csh setup pipe_tasks -t v9_2 cd /location/to/notebook ipython notebook

Next up, download an image from DECam, which has already been processed through the LSST DM pipeline.

In [1]:

%%bash
if [ -f calexp-0283453_10.fits ];
then
    echo "File calexp-0283453_10.fits exists"
else
    echo "wget http://www.astro.washington.edu/users/becker/tmp/calexp-0283453_10.fits"
    wget http://www.astro.washington.edu/users/becker/tmp/calexp-0283453_10.fits
fi

File calexp-0283453_10.fits exists

For the next several cells, we outline the process of building up a photometric pipeline from a series of Python commands. In the real world, these code blocks would exist in a single script that you would call on the command line (e.g. like SExtractor). However, for the purposes of this Data School we break this script down into distinct logical units.

We first import all the important functionality that we will need, most of it from LSST (and visible to Python when you "set up" the package):

In [2]:

import sys
import numpy as np

import lsst.pex.policy as pexPolicy
import lsst.afw.detection as afwDetection
import lsst.afw.geom as afwGeom
import lsst.afw.image as afwImage
import lsst.afw.math as afwMath
import lsst.afw.table as afwTable
import lsst.afw.display.ds9 as ds9
import lsst.afw.display.utils as displayUtils
import lsst.meas.algorithms as measAlg

Read in the image we just downloaded as an LSST Exposure object, and display its image to the screen using ds9. If you get a complaint that "ds9: Command not found", you need to make sure that the command "ds9" is in your path when you run the "ipython notebook" command (i.e. the command "which ds9" returns an answer).

In [3]:

fileName = "calexp-0283453_10.fits"
exposure = afwImage.ExposureF(fileName)
mi = exposure.getMaskedImage()
im = mi.getImage()
frame = 1
ds9.mtv(exposure, frame=frame, title="Original Image Off of Disk"); frame += 1

The blue masks show where the LSST pipeline previously detected sources. Since we want to re-do this analysis, clear the mask planes that indicate a source has been detected, and then redisplay the image.

In [4]:

mask = mi.getMask()
mask &= ~(mask.getPlaneBitMask("DETECTED")) 
ds9.mtv(exposure, frame=frame, title="Detection Mask Removed"); frame += 1

Note that some, but not all, of the masks have been removed. E.g. the saturated star at (603,765) remains masked.

We next desire to subtract the background off of the image, so that we can detect peaks above the background. We set up a grid of 64x64 pixel areas, compute moments of the pixels within each bin (which should be dominated by sky pixels), and fit a smooth function to the values within each bin. This function represents the background in the image, which we then subtract off.

In [5]:

#
# Subtract background
#
back_size = 64
bctrl = afwMath.BackgroundControl(im.getWidth()//back_size  + 1,
                                  im.getHeight()//back_size + 1)
backobj = afwMath.makeBackground(im, bctrl)
im -= backobj.getImageF("LINEAR")
ds9.mtv(exposure, frame=frame, title="Background Removed"); frame += 1

Note that the image basically looks the same as in the previous image, but the background values are centered on zero.

Next, we look at the statistics of image, which are dominated by the sky pixels. We look for the clipped mean (by clipping, we remove the high-sigma and low-sigma outliers, with the expectation that removing the high-sigma outliers removes the pixels that are associated with bright sources). We define an upper threshold for detection of

$mean + 5 \times standard\ deviaton$

In [6]:

nsigma = 5
stats = afwMath.makeStatistics(im, afwMath.MEANCLIP | afwMath.STDEVCLIP)
threshold = stats.getValue(afwMath.MEANCLIP) + nsigma*stats.getValue(afwMath.STDEVCLIP)
print "Mean of the image is %.3f counts, stdev is %.3f counts" % (stats.getValue(afwMath.MEANCLIP), stats.getValue(afwMath.STDEVCLIP))
print "The %d-sigma detection threshold is at %.2f counts" % (nsigma, threshold)

Mean of the image is -0.041 counts, stdev is 10.842 counts
The 5-sigma detection threshold is at 54.17 counts

We next want to look for objects within this image. Formally, the optimal filter for detecting signals of shape g(x) from within an image $I(x)=A*g(x) + \epsilon(x)$ is g(x). In other words, the optimal filter for g(x) is g(x) itself! For the detection of point sources, g(x) is the Psf of the image.

However, at this point in time we don't know the Psf. So we use an approximation to the Psf; something that is sharply peaked and which falls off with some charateristic width. Here we use a filter with a FWHM = 2 pixels. We smooth the image with this filter, which yields a maximum likelihood image for the detection of point sources that have a width of 2 pixels.

In [7]:

#
# Smooth image with a
#    1 2 1
#    2 4 2
#    1 2 1
# filter
oneD = afwMath.PolynomialFunction1D([2, 0, -1])
kernel = afwMath.SeparableKernel(3, 3, oneD, oneD)
smoothedIm = im.Factory(im.getDimensions())
afwMath.convolve(smoothedIm, im, kernel)
ds9.mtv(smoothedIm, frame=frame, title="Detection Image (smoothed with a filter)"); frame += 1

Note that this image looks smoothed compared to the previous image; this is the effect of the convolution.

Also note that we calculate the statistics for detection from the original image, but will perform detection on this smoothed image.

We then search for these 5-sigma peaks within the image, and group together nearby pixels that are also above this threshold. These are stored in "footprints" which represent an associated set of pixels.

In [8]:

npixMin = 1
fs0 = afwDetection.FootprintSet(smoothedIm, afwDetection.Threshold(threshold), npixMin)
ds9.mtv(smoothedIm, frame=frame, title="Detected Footprints at %d sigma" % (nsigma))
for footprint in fs0.getFootprints():
    displayUtils.drawFootprint(footprint, frame=frame)
frame += 1

Because we are only associating 5-sigma pixels with each other, grow these detections a little bit to include the lower-sigma (e.g. 3-sigma) pixels within the footprint.

In [9]:

grow = 1
isotropic = False
fs = afwDetection.FootprintSet(fs0, grow, isotropic)
ds9.mtv(smoothedIm, frame=frame, title="Grown Footprints")
for footprint in fs.getFootprints():
    displayUtils.drawFootprint(footprint, frame=frame)
frame += 1

Now tell the original image that we have "detected" footprints at these positions. The DETECTION plane of the image will now be set at these pixel locations.

In [10]:

fs.setMask(mi.getMask(), "DETECTED")
ds9.mtv(mi, frame=frame, title="Detection plane in original image"); frame += 1

Next, we want to perform measurements on each footprint. This subset of measurements includes zeroth moments (centroids), first moments (flux), second moments (shape).

In [11]:

# Define the measurements we want to make
apRad = 3
ctrlCentroid = measAlg.SdssCentroidControl()
ctrlAperture = measAlg.SincFluxControl()
ctrlAperture.radius2 = apRad

# Make a source table schema that is designed to accept the measurement results from each algorithm
schema = afwTable.SourceTable.makeMinimalSchema()
algorithms = [
    measAlg.MeasureSourcesBuilder().addAlgorithm(ctrlCentroid).build(schema),
    measAlg.MeasureSourcesBuilder().addAlgorithm(ctrlAperture).build(schema)]
    
# Build an empty source catalog from this schema
cat = afwTable.SourceCatalog(schema)
table = cat.table
table.defineCentroid("centroid.sdss")
table.defineApFlux("flux.sinc")

# Turn the Footprints into Sources 
fs.makeSources(cat)
print "Measuring %d objects" % (len(cat))

# And apply each algorithm to each Source, measuring on the orignal image
for source in cat:
    for alg in algorithms:
        alg.apply(source, exposure)

Measuring 492 objects

Finally, display the position of each measured source, indicating with "plus" signs the measurements.

In [12]:

ds9.mtv(exposure, frame=frame, title="Measured Sources")
with ds9.Buffering():
    for source in cat:
        ds9.dot("+", *source.getCentroid(), frame=frame)
frame += 1        

Can we do better?¶

Yes we can! Now that we have measured sources in the image, we can use them to build a Psf model for the image. This requires separating the stars from the galaxies, and only building the Psf using the stars. Once we have this Psf, which again is the optimal filter for point source detection, we smooth the image using this filter, re-detect, and re-measure the sources. Ideally this should give us fewer noise detections of noise and optimal detection depth.¶

Many of the details below are bookkeeping and unimportant for understanding conceptually the details of the algorithm. What we will do is look at the results of the processing to understand what has happened...

In [13]:

from lsst.meas.algorithms.detection import SourceDetectionTask
from lsst.meas.algorithms.measurement import SourceMeasurementTask
from lsst.pipe.tasks.measurePsf import MeasurePsfTask

schema = afwTable.SourceTable.makeMinimalSchema()

config = SourceDetectionTask.ConfigClass()
config.reEstimateBackground = False
detectionTask = SourceDetectionTask(config=config, schema=schema)
 
config = SourceMeasurementTask.ConfigClass()
config.slots.psfFlux = "flux.sinc" # use of the psf flux is hardcoded in secondMomentStarSelector
measureTask = SourceMeasurementTask(schema, config=config)

config = MeasurePsfTask.ConfigClass()
starSelector = config.starSelector.apply()
starSelector.config.badFlags = ["flags.pixel.edge", "flags.pixel.cr.center", "flags.pixel.interpolated.center", "flags.pixel.saturated.center"]
psfDeterminer = config.psfDeterminer.apply()
psfDeterminer.config.sizeCellX = 128
psfDeterminer.config.sizeCellY = 128
psfDeterminer.config.spatialOrder = 1
psfDeterminer.config.nEigenComponents = 3
measurePsfTask = MeasurePsfTask(config=config, schema=schema)

# Again clear the detection masks
mask = exposure.getMaskedImage().getMask()
mask &= ~(mask.getPlaneBitMask("DETECTED")) 

# And then re-detect, re-measure, select stars, and build the Psf model
table = afwTable.SourceTable.make(schema)
sources = detectionTask.run(table, exposure, sigma=3).sources
measureTask.measure(exposure, sources)
result = measurePsfTask.run(exposure, sources)

First, visualize what does the Psf look like?

In [14]:

psf = result.psf
psfIm = psf.computeImage()
ds9.mtv(psfIm, frame=frame, title="Psf Image"); frame += 1

Look at how it varies across the image by rendering it in a grid:

In [15]:

import lsst.meas.algorithms.utils as measUtils
cellSet = result.cellSet
measUtils.showPsfMosaic(exposure, psf=psf, frame=frame); frame += 1

What objects were identified as stars and used in the Psf model? Mark in YELLOW if they were candidates for stars, and mark in RED if they were actually used in the Psf fitting.

In [16]:

ds9.mtv(exposure, frame=frame, title="Re-Measured Footprints")
with ds9.Buffering():
    for s in sources:
        xy = s.getCentroid()
        ds9.dot('+', *xy, frame=frame)
        if s.get("calib.psf.candidate"):
            ds9.dot('x', *xy, ctype=ds9.YELLOW, frame=frame)
        if s.get("calib.psf.used"):
            ds9.dot('o', *xy, size=4, ctype=ds9.RED, frame=frame)
frame += 1

Finally, repeat the process of smoothing, detection, and measurement but using the actual Psf.

However, because the Psf is a complex function (a 2-dimensional shape that varies spatially) convolutions with an actual Psf are sloooooooww.... Instead, we create a pure Gaussian with the width of the Psf. The convolution ends up being far faster, with the loss of some accuracy.

In [17]:

shape = psf.computeShape()
sigma = shape.getDeterminantRadius()
print "Psf approximate Gaussian sigma = %.3f pixels; FWHM = %.3f pixels" % (sigma, sigma * 2. * np.sqrt(2. * np.log(2.)))
kWidth = (int(sigma * 7 + 0.5) // 2) * 2 + 1 # make sure kernel size is odd
gaussFunc = afwMath.GaussianFunction1D(sigma)
gaussKernel = afwMath.SeparableKernel(kWidth, kWidth, gaussFunc, gaussFunc)

im = exposure.getMaskedImage().getImage()
smoothedIm = im.Factory(im.getDimensions())
afwMath.convolve(smoothedIm, im, gaussKernel)
ds9.mtv(smoothedIm, frame=frame, title="Detection Image (smoothed with appx Psf)"); frame += 1

Psf approximate Gaussian sigma = 1.956 pixels; FWHM = 4.606 pixels

With the new smoothed image, use the same threshold as defined above to find footprints

In [18]:

fs0 = afwDetection.FootprintSet(smoothedIm, afwDetection.Threshold(threshold), npixMin)
fs = afwDetection.FootprintSet(fs0, grow, isotropic)
ds9.mtv(smoothedIm, frame=frame, title="Re-detected and re-grown Footprints")
for footprint in fs.getFootprints():
    displayUtils.drawFootprint(footprint, frame=frame)
frame += 1

And finally, remeasure the detected footprints as a new set of sources

In [19]:

# Build a new, empty source catalog from this schema
cat = afwTable.SourceCatalog(schema)
table = cat.table
table.defineCentroid("centroid.sdss")
table.defineApFlux("flux.sinc")

# Turn the Footprints into Sources 
fs.makeSources(cat)
print "Measuring %d objects" % (len(cat))

# And apply each algorithm to each Source, measuring on the orignal image
for source in cat:
    for alg in algorithms:
        alg.apply(source, exposure)

Measuring 319 objects

Doing the same thing in SExtractor/PSFEx¶

Download from: http://www.astromatic.net/software/sextractor http://www.astromatic.net/software/psfex Or use MacPorts: sudo port install sextractor (might need to add "+gcc48") sudo port install psfex (might need to add "+gcc48")

In [20]:

%%bash
sex -d

# Default configuration file for SExtractor 2.19.5
# EB 2014-06-07
#
 
#-------------------------------- Catalog ------------------------------------
 
CATALOG_NAME     test.cat       # name of the output catalog
CATALOG_TYPE     ASCII_HEAD     # NONE,ASCII,ASCII_HEAD, ASCII_SKYCAT,
                                # ASCII_VOTABLE, FITS_1.0 or FITS_LDAC
PARAMETERS_NAME  default.param  # name of the file containing catalog contents
 
#------------------------------- Extraction ----------------------------------
 
DETECT_TYPE      CCD            # CCD (linear) or PHOTO (with gamma correction)
DETECT_MINAREA   5              # min. # of pixels above threshold
DETECT_THRESH    1.5            # <sigmas> or <threshold>,<ZP> in mag.arcsec-2
ANALYSIS_THRESH  1.5            # <sigmas> or <threshold>,<ZP> in mag.arcsec-2
 
FILTER           Y              # apply filter for detection (Y or N)?
FILTER_NAME      default.conv   # name of the file containing the filter
 
DEBLEND_NTHRESH  32             # Number of deblending sub-thresholds
DEBLEND_MINCONT  0.005          # Minimum contrast parameter for deblending
 
CLEAN            Y              # Clean spurious detections? (Y or N)?
CLEAN_PARAM      1.0            # Cleaning efficiency
 
MASK_TYPE        CORRECT        # type of detection MASKing: can be one of
                                # NONE, BLANK or CORRECT
 
#------------------------------ Photometry -----------------------------------
 
PHOT_APERTURES   5              # MAG_APER aperture diameter(s) in pixels
PHOT_AUTOPARAMS  2.5, 3.5       # MAG_AUTO parameters: <Kron_fact>,<min_radius>
PHOT_PETROPARAMS 2.0, 3.5       # MAG_PETRO parameters: <Petrosian_fact>,
                                # <min_radius>
 
SATUR_LEVEL      50000.0        # level (in ADUs) at which arises saturation
SATUR_KEY        SATURATE       # keyword for saturation level (in ADUs)
 
MAG_ZEROPOINT    0.0            # magnitude zero-point
MAG_GAMMA        4.0            # gamma of emulsion (for photographic scans)
GAIN             0.0            # detector gain in e-/ADU
GAIN_KEY         GAIN           # keyword for detector gain in e-/ADU
PIXEL_SCALE      1.0            # size of pixel in arcsec (0=use FITS WCS info)
 
#------------------------- Star/Galaxy Separation ----------------------------
 
SEEING_FWHM      1.2            # stellar FWHM in arcsec
STARNNW_NAME     default.nnw    # Neural-Network_Weight table filename
 
#------------------------------ Background -----------------------------------
 
BACK_SIZE        64             # Background mesh: <size> or <width>,<height>
BACK_FILTERSIZE  3              # Background filter: <size> or <width>,<height>
 
BACKPHOTO_TYPE   GLOBAL         # can be GLOBAL or LOCAL
 
#------------------------------ Check Image ----------------------------------
 
CHECKIMAGE_TYPE  NONE           # can be NONE, BACKGROUND, BACKGROUND_RMS,
                                # MINIBACKGROUND, MINIBACK_RMS, -BACKGROUND,
                                # FILTERED, OBJECTS, -OBJECTS, SEGMENTATION,
                                # or APERTURES
CHECKIMAGE_NAME  check.fits     # Filename for the check-image
 
#--------------------- Memory (change with caution!) -------------------------
 
MEMORY_OBJSTACK  3000           # number of objects in stack
MEMORY_PIXSTACK  300000         # number of pixels in stack
MEMORY_BUFSIZE   1024           # number of lines in buffer
 
#----------------------------- Miscellaneous ---------------------------------
 
VERBOSE_TYPE     NORMAL         # can be QUIET, NORMAL or FULL
HEADER_SUFFIX    .head          # Filename extension for additional headers
WRITE_XML        N              # Write XML file (Y/N)?
XML_NAME         sex.xml        # Filename for XML output

In [21]:

%%bash --err error
cp /opt/local/share/sextractor/default.param .
cp /opt/local/share/sextractor/default.conv .
cp /opt/local/share/sextractor/default.sex .

cat default.param | grep -v ^\#
sex -PARAMETERS_NAME default.param -FILTER_NAME default.conv -CATALOG_TYPE FITS_LDAC -CATALOG_NAME calexp-0283453_10.ldac "calexp-0283453_10.fits[1]" 
sex -PARAMETERS_NAME default.param -FILTER_NAME default.conv -CATALOG_TYPE ASCII_HEAD -CATALOG_NAME calexp-0283453_10.cat "calexp-0283453_10.fits[1]"

FLUX_APER                #Flux vector within fixed circular aperture(s)             [count]
FLUXERR_APER             #RMS error vector for aperture flux(es)                    [count]
FLUX_MAX                  Peak flux above background                                [count]
X_IMAGE                   Object position along x                                   [pixel]
Y_IMAGE                   Object position along y                                   [pixel]
FLAGS                     Extraction flags                                         
ELONGATION                A_IMAGE/B_IMAGE                                          
VIGNET(10,10)            #Pixel data around detection                               [count]
FLUX_RADIUS(1)           #Fraction-of-light radii                                   [pixel]
SNR_WIN

In [22]:

print error

> 
----- SExtractor 2.19.5 started on 2014-08-17 at 09:04:15 with 1 thread

> Setting catalog parameters
> Reading detection filter
> Initializing catalog
> Looking for calexp-0283453_10.fits
----- Measuring from: calexp-0283453_10.fits [1/9]
      "Unnamed" / no ext. header / 2046x4094 / 32 bits (floats)
Detection+Measurement image: > Setting up background maps
> Setting up background map at line:   64
> Setting up background map at line:  128
> Setting up background map at line:  192
> Setting up background map at line:  256
> Setting up background map at line:  320
> Setting up background map at line:  384
> Setting up background map at line:  448
> Setting up background map at line:  512
> Setting up background map at line:  576
> Setting up background map at line:  640
> Setting up background map at line:  704
> Setting up background map at line:  768
> Setting up background map at line:  832
> Setting up background map at line:  896
> Setting up background map at line:  960
> Setting up background map at line: 1024
> Setting up background map at line: 1088
> Setting up background map at line: 1152
> Setting up background map at line: 1216
> Setting up background map at line: 1280
> Setting up background map at line: 1344
> Setting up background map at line: 1408
> Setting up background map at line: 1472
> Setting up background map at line: 1536
> Setting up background map at line: 1600
> Setting up background map at line: 1664
> Setting up background map at line: 1728
> Setting up background map at line: 1792
> Setting up background map at line: 1856
> Setting up background map at line: 1920
> Setting up background map at line: 1984
> Setting up background map at line: 2048
> Setting up background map at line: 2112
> Setting up background map at line: 2176
> Setting up background map at line: 2240
> Setting up background map at line: 2304
> Setting up background map at line: 2368
> Setting up background map at line: 2432
> Setting up background map at line: 2496
> Setting up background map at line: 2560
> Setting up background map at line: 2624
> Setting up background map at line: 2688
> Setting up background map at line: 2752
> Setting up background map at line: 2816
> Setting up background map at line: 2880
> Setting up background map at line: 2944
> Setting up background map at line: 3008
> Setting up background map at line: 3072
> Setting up background map at line: 3136
> Setting up background map at line: 3200
> Setting up background map at line: 3264
> Setting up background map at line: 3328
> Setting up background map at line: 3392
> Setting up background map at line: 3456
> Setting up background map at line: 3520
> Setting up background map at line: 3584
> Setting up background map at line: 3648
> Setting up background map at line: 3712
> Setting up background map at line: 3776
> Setting up background map at line: 3840
> Setting up background map at line: 3904
> Setting up background map at line: 3968
> Setting up background map at line: 4032
> Filtering background map(s)
> Computing background d-map
> Computing background-noise d-map
(M+D) Background: 404.979    RMS: 10.791     / Threshold: 53.9551    
> Scanning image
> Line:   25  Objects:        1 detected /        0 sextracted
> Line:   50  Objects:       10 detected /        0 sextracted
> Line:   75  Objects:       11 detected /        0 sextracted
> Line:  100  Objects:       19 detected /        0 sextracted
> Line:  125  Objects:       26 detected /        0 sextracted
> Line:  150  Objects:       29 detected /        0 sextracted
> Line:  175  Objects:       31 detected /        0 sextracted
> Line:  200  Objects:       39 detected /        0 sextracted
> Line:  225  Objects:       45 detected /        0 sextracted
> Line:  250  Objects:       50 detected /        0 sextracted
> Line:  275  Objects:       55 detected /        0 sextracted
> Line:  300  Objects:       60 detected /        0 sextracted
> Line:  325  Objects:       61 detected /        0 sextracted
> Line:  350  Objects:       67 detected /        0 sextracted
> Line:  375  Objects:       67 detected /        0 sextracted
> Line:  400  Objects:       73 detected /        0 sextracted
> Line:  425  Objects:       76 detected /        0 sextracted
> Line:  450  Objects:       80 detected /        0 sextracted
> Line:  475  Objects:       81 detected /        0 sextracted
> Line:  500  Objects:       87 detected /        0 sextracted
> Line:  525  Objects:       90 detected /        0 sextracted
> Line:  550  Objects:       91 detected /        0 sextracted
> Line:  575  Objects:       94 detected /        0 sextracted
> Line:  600  Objects:       97 detected /        0 sextracted
> Line:  625  Objects:      100 detected /        0 sextracted
> Line:  650  Objects:      107 detected /        0 sextracted
> Line:  675  Objects:      109 detected /        0 sextracted
> Line:  700  Objects:      115 detected /        0 sextracted
> Line:  725  Objects:      117 detected /        0 sextracted
> Line:  750  Objects:      124 detected /        0 sextracted
> Line:  775  Objects:      129 detected /        0 sextracted
> Line:  800  Objects:      132 detected /        0 sextracted
> Line:  825  Objects:      137 detected /        0 sextracted
> Line:  850  Objects:      141 detected /        0 sextracted
> Line:  875  Objects:      144 detected /        0 sextracted
> Line:  900  Objects:      156 detected /        0 sextracted
> Line:  925  Objects:      158 detected /        0 sextracted
> Line:  950  Objects:      161 detected /        0 sextracted
> Line:  975  Objects:      161 detected /        0 sextracted
> Line: 1000  Objects:      167 detected /        0 sextracted
> Line: 1022  Objects:      168 detected /        0 sextracted
> Line: 1025  Objects:      170 detected /        6 sextracted
> Line: 1050  Objects:      171 detected /       12 sextracted
> Line: 1075  Objects:      173 detected /       16 sextracted
> Line: 1100  Objects:      182 detected /       19 sextracted
> Line: 1125  Objects:      187 detected /       25 sextracted
> Line: 1150  Objects:      188 detected /       28 sextracted
> Line: 1175  Objects:      194 detected /       30 sextracted
> Line: 1200  Objects:      200 detected /       37 sextracted
> Line: 1225  Objects:      208 detected /       40 sextracted
> Line: 1250  Objects:      211 detected /       42 sextracted
> Line: 1275  Objects:      212 detected /       47 sextracted
> Line: 1300  Objects:      213 detected /       50 sextracted
> Line: 1325  Objects:      213 detected /       52 sextracted
> Line: 1350  Objects:      217 detected /       54 sextracted
> Line: 1375  Objects:      231 detected /       56 sextracted
> Line: 1400  Objects:      239 detected /       60 sextracted
> Line: 1425  Objects:      246 detected /       63 sextracted
> Line: 1450  Objects:      248 detected /       66 sextracted
> Line: 1475  Objects:      250 detected /       67 sextracted
> Line: 1500  Objects:      253 detected /       75 sextracted
> Line: 1525  Objects:      260 detected /       79 sextracted
> Line: 1550  Objects:      262 detected /       79 sextracted
> Line: 1575  Objects:      274 detected /       83 sextracted
> Line: 1600  Objects:      276 detected /       87 sextracted
> Line: 1625  Objects:      282 detected /       90 sextracted
> Line: 1650  Objects:      285 detected /       91 sextracted
> Line: 1675  Objects:      286 detected /       93 sextracted
> Line: 1700  Objects:      290 detected /       98 sextracted
> Line: 1725  Objects:      293 detected /      101 sextracted
> Line: 1750  Objects:      293 detected /      106 sextracted
> Line: 1775  Objects:      297 detected /      115 sextracted
> Line: 1800  Objects:      298 detected /      121 sextracted
> Line: 1825  Objects:      303 detected /      122 sextracted
> Line: 1850  Objects:      316 detected /      125 sextracted
> Line: 1875  Objects:      320 detected /      128 sextracted
> Line: 1900  Objects:      320 detected /      131 sextracted
> Line: 1925  Objects:      321 detected /      132 sextracted
> Line: 1950  Objects:      323 detected /      135 sextracted
> Line: 1975  Objects:      329 detected /      139 sextracted
> Line: 2000  Objects:      339 detected /      142 sextracted
> Line: 2025  Objects:      340 detected /      147 sextracted
> Line: 2050  Objects:      353 detected /      151 sextracted
> Line: 2075  Objects:      356 detected /      152 sextracted
> Line: 2100  Objects:      357 detected /      156 sextracted
> Line: 2125  Objects:      368 detected /      161 sextracted
> Line: 2150  Objects:      379 detected /      162 sextracted
> Line: 2175  Objects:      381 detected /      166 sextracted
> Line: 2200  Objects:      392 detected /      169 sextracted
> Line: 2225  Objects:      394 detected /      172 sextracted
> Line: 2250  Objects:      403 detected /      175 sextracted
> Line: 2275  Objects:      408 detected /      178 sextracted
> Line: 2300  Objects:      409 detected /      180 sextracted
> Line: 2325  Objects:      410 detected /      181 sextracted
> Line: 2350  Objects:      410 detected /      184 sextracted
> Line: 2375  Objects:      415 detected /      187 sextracted
> Line: 2400  Objects:      417 detected /      192 sextracted
> Line: 2425  Objects:      418 detected /      196 sextracted
> Line: 2450  Objects:      424 detected /      198 sextracted
> Line: 2475  Objects:      428 detected /      203 sextracted
> Line: 2500  Objects:      432 detected /      204 sextracted
> Line: 2525  Objects:      435 detected /      205 sextracted
> Line: 2550  Objects:      440 detected /      207 sextracted
> Line: 2575  Objects:      452 detected /      208 sextracted
> Line: 2600  Objects:      460 detected /      215 sextracted
> Line: 2625  Objects:      462 detected /      218 sextracted
> Line: 2650  Objects:      465 detected /      220 sextracted
> Line: 2675  Objects:      468 detected /      222 sextracted
> Line: 2700  Objects:      471 detected /      225 sextracted
> Line: 2725  Objects:      477 detected /      229 sextracted
> Line: 2750  Objects:      481 detected /      231 sextracted
> Line: 2775  Objects:      484 detected /      236 sextracted
> Line: 2800  Objects:      484 detected /      237 sextracted
> Line: 2825  Objects:      487 detected /      240 sextracted
> Line: 2850  Objects:      493 detected /      243 sextracted
> Line: 2875  Objects:      495 detected /      245 sextracted
> Line: 2900  Objects:      497 detected /      248 sextracted
> Line: 2925  Objects:      504 detected /      255 sextracted
> Line: 2950  Objects:      510 detected /      258 sextracted
> Line: 2975  Objects:      511 detected /      265 sextracted
> Line: 3000  Objects:      511 detected /      268 sextracted
> Line: 3025  Objects:      511 detected /      270 sextracted
> Line: 3050  Objects:      512 detected /      270 sextracted
> Line: 3075  Objects:      517 detected /      270 sextracted
> Line: 3100  Objects:      518 detected /      275 sextracted
> Line: 3125  Objects:      521 detected /      285 sextracted
> Line: 3150  Objects:      526 detected /      295 sextracted
> Line: 3175  Objects:      532 detected /      295 sextracted
> Line: 3200  Objects:      533 detected /      296 sextracted
> Line: 3225  Objects:      535 detected /      297 sextracted
> Line: 3250  Objects:      538 detected /      301 sextracted
> Line: 3275  Objects:      540 detected /      303 sextracted
> Line: 3300  Objects:      540 detected /      304 sextracted
> Line: 3325  Objects:      541 detected /      305 sextracted
> Line: 3350  Objects:      545 detected /      306 sextracted
> Line: 3375  Objects:      547 detected /      309 sextracted
> Line: 3400  Objects:      548 detected /      312 sextracted
> Line: 3425  Objects:      550 detected /      312 sextracted
> Line: 3450  Objects:      556 detected /      315 sextracted
> Line: 3475  Objects:      557 detected /      316 sextracted
> Line: 3500  Objects:      566 detected /      321 sextracted
> Line: 3525  Objects:      568 detected /      324 sextracted
> Line: 3550  Objects:      571 detected /      332 sextracted
> Line: 3575  Objects:      587 detected /      338 sextracted
> Line: 3600  Objects:      594 detected /      342 sextracted
> Line: 3625  Objects:      600 detected /      348 sextracted
> Line: 3650  Objects:      605 detected /      350 sextracted
> Line: 3675  Objects:      607 detected /      355 sextracted
> Line: 3700  Objects:      609 detected /      357 sextracted
> Line: 3725  Objects:      613 detected /      364 sextracted
> Line: 3750  Objects:      615 detected /      368 sextracted
> Line: 3775  Objects:      616 detected /      370 sextracted
> Line: 3800  Objects:      619 detected /      370 sextracted
> Line: 3825  Objects:      640 detected /      384 sextracted
> Line: 3850  Objects:      642 detected /      389 sextracted
> Line: 3875  Objects:      646 detected /      392 sextracted
> Line: 3900  Objects:      648 detected /      395 sextracted
> Line: 3922  Objects:      650 detected /      400 sextracted
> Line: 3925  Objects:      650 detected /      401 sextracted
> Line: 3950  Objects:      656 detected /      406 sextracted
> Line: 3975  Objects:      657 detected /      409 sextracted
> Line: 4000  Objects:      660 detected /      409 sextracted
> Line: 4025  Objects:      665 detected /      410 sextracted
> Line: 4050  Objects:      672 detected /      410 sextracted
> Line: 4075  Objects:      681 detected /      415 sextracted
      Objects: detected 686      / sextracted 552             

> Closing files
> 
> All done (in 1.6 s: 2532.4 lines/s , 341.4 detections/s)
> 
----- SExtractor 2.19.5 started on 2014-08-17 at 09:04:16 with 1 thread

> Setting catalog parameters
> Reading detection filter
> Initializing catalog
> Looking for calexp-0283453_10.fits
----- Measuring from: calexp-0283453_10.fits [1/9]
      "Unnamed" / no ext. header / 2046x4094 / 32 bits (floats)
Detection+Measurement image: > Setting up background maps
> Setting up background map at line:   64
> Setting up background map at line:  128
> Setting up background map at line:  192
> Setting up background map at line:  256
> Setting up background map at line:  320
> Setting up background map at line:  384
> Setting up background map at line:  448
> Setting up background map at line:  512
> Setting up background map at line:  576
> Setting up background map at line:  640
> Setting up background map at line:  704
> Setting up background map at line:  768
> Setting up background map at line:  832
> Setting up background map at line:  896
> Setting up background map at line:  960
> Setting up background map at line: 1024
> Setting up background map at line: 1088
> Setting up background map at line: 1152
> Setting up background map at line: 1216
> Setting up background map at line: 1280
> Setting up background map at line: 1344
> Setting up background map at line: 1408
> Setting up background map at line: 1472
> Setting up background map at line: 1536
> Setting up background map at line: 1600
> Setting up background map at line: 1664
> Setting up background map at line: 1728
> Setting up background map at line: 1792
> Setting up background map at line: 1856
> Setting up background map at line: 1920
> Setting up background map at line: 1984
> Setting up background map at line: 2048
> Setting up background map at line: 2112
> Setting up background map at line: 2176
> Setting up background map at line: 2240
> Setting up background map at line: 2304
> Setting up background map at line: 2368
> Setting up background map at line: 2432
> Setting up background map at line: 2496
> Setting up background map at line: 2560
> Setting up background map at line: 2624
> Setting up background map at line: 2688
> Setting up background map at line: 2752
> Setting up background map at line: 2816
> Setting up background map at line: 2880
> Setting up background map at line: 2944
> Setting up background map at line: 3008
> Setting up background map at line: 3072
> Setting up background map at line: 3136
> Setting up background map at line: 3200
> Setting up background map at line: 3264
> Setting up background map at line: 3328
> Setting up background map at line: 3392
> Setting up background map at line: 3456
> Setting up background map at line: 3520
> Setting up background map at line: 3584
> Setting up background map at line: 3648
> Setting up background map at line: 3712
> Setting up background map at line: 3776
> Setting up background map at line: 3840
> Setting up background map at line: 3904
> Setting up background map at line: 3968
> Setting up background map at line: 4032
> Filtering background map(s)
> Computing background d-map
> Computing background-noise d-map
(M+D) Background: 404.979    RMS: 10.791     / Threshold: 53.9551    
> Scanning image
> Line:   25  Objects:        1 detected /        0 sextracted
> Line:   50  Objects:       10 detected /        0 sextracted
> Line:   75  Objects:       11 detected /        0 sextracted
> Line:  100  Objects:       19 detected /        0 sextracted
> Line:  125  Objects:       26 detected /        0 sextracted
> Line:  150  Objects:       29 detected /        0 sextracted
> Line:  175  Objects:       31 detected /        0 sextracted
> Line:  200  Objects:       39 detected /        0 sextracted
> Line:  225  Objects:       45 detected /        0 sextracted
> Line:  250  Objects:       50 detected /        0 sextracted
> Line:  275  Objects:       55 detected /        0 sextracted
> Line:  300  Objects:       60 detected /        0 sextracted
> Line:  325  Objects:       61 detected /        0 sextracted
> Line:  350  Objects:       67 detected /        0 sextracted
> Line:  375  Objects:       67 detected /        0 sextracted
> Line:  400  Objects:       73 detected /        0 sextracted
> Line:  425  Objects:       76 detected /        0 sextracted
> Line:  450  Objects:       80 detected /        0 sextracted
> Line:  475  Objects:       81 detected /        0 sextracted
> Line:  500  Objects:       87 detected /        0 sextracted
> Line:  525  Objects:       90 detected /        0 sextracted
> Line:  550  Objects:       91 detected /        0 sextracted
> Line:  575  Objects:       94 detected /        0 sextracted
> Line:  600  Objects:       97 detected /        0 sextracted
> Line:  625  Objects:      100 detected /        0 sextracted
> Line:  650  Objects:      107 detected /        0 sextracted
> Line:  675  Objects:      109 detected /        0 sextracted
> Line:  700  Objects:      115 detected /        0 sextracted
> Line:  725  Objects:      117 detected /        0 sextracted
> Line:  750  Objects:      124 detected /        0 sextracted
> Line:  775  Objects:      129 detected /        0 sextracted
> Line:  800  Objects:      132 detected /        0 sextracted
> Line:  825  Objects:      137 detected /        0 sextracted
> Line:  850  Objects:      141 detected /        0 sextracted
> Line:  875  Objects:      144 detected /        0 sextracted
> Line:  900  Objects:      156 detected /        0 sextracted
> Line:  925  Objects:      158 detected /        0 sextracted
> Line:  950  Objects:      161 detected /        0 sextracted
> Line:  975  Objects:      161 detected /        0 sextracted
> Line: 1000  Objects:      167 detected /        0 sextracted
> Line: 1022  Objects:      168 detected /        0 sextracted
> Line: 1025  Objects:      170 detected /        6 sextracted
> Line: 1050  Objects:      171 detected /       12 sextracted
> Line: 1075  Objects:      173 detected /       16 sextracted
> Line: 1100  Objects:      182 detected /       19 sextracted
> Line: 1125  Objects:      187 detected /       25 sextracted
> Line: 1150  Objects:      188 detected /       28 sextracted
> Line: 1175  Objects:      194 detected /       30 sextracted
> Line: 1200  Objects:      200 detected /       37 sextracted
> Line: 1225  Objects:      208 detected /       40 sextracted
> Line: 1250  Objects:      211 detected /       42 sextracted
> Line: 1275  Objects:      212 detected /       47 sextracted
> Line: 1300  Objects:      213 detected /       50 sextracted
> Line: 1325  Objects:      213 detected /       52 sextracted
> Line: 1350  Objects:      217 detected /       54 sextracted
> Line: 1375  Objects:      231 detected /       56 sextracted
> Line: 1400  Objects:      239 detected /       60 sextracted
> Line: 1425  Objects:      246 detected /       63 sextracted
> Line: 1450  Objects:      248 detected /       66 sextracted
> Line: 1475  Objects:      250 detected /       67 sextracted
> Line: 1500  Objects:      253 detected /       75 sextracted
> Line: 1525  Objects:      260 detected /       79 sextracted
> Line: 1550  Objects:      262 detected /       79 sextracted
> Line: 1575  Objects:      274 detected /       83 sextracted
> Line: 1600  Objects:      276 detected /       87 sextracted
> Line: 1625  Objects:      282 detected /       90 sextracted
> Line: 1650  Objects:      285 detected /       91 sextracted
> Line: 1675  Objects:      286 detected /       93 sextracted
> Line: 1700  Objects:      290 detected /       98 sextracted
> Line: 1725  Objects:      293 detected /      101 sextracted
> Line: 1750  Objects:      293 detected /      106 sextracted
> Line: 1775  Objects:      297 detected /      115 sextracted
> Line: 1800  Objects:      298 detected /      121 sextracted
> Line: 1825  Objects:      303 detected /      122 sextracted
> Line: 1850  Objects:      316 detected /      125 sextracted
> Line: 1875  Objects:      320 detected /      128 sextracted
> Line: 1900  Objects:      320 detected /      131 sextracted
> Line: 1925  Objects:      321 detected /      132 sextracted
> Line: 1950  Objects:      323 detected /      135 sextracted
> Line: 1975  Objects:      329 detected /      139 sextracted
> Line: 2000  Objects:      339 detected /      142 sextracted
> Line: 2025  Objects:      340 detected /      147 sextracted
> Line: 2050  Objects:      353 detected /      151 sextracted
> Line: 2075  Objects:      356 detected /      152 sextracted
> Line: 2100  Objects:      357 detected /      156 sextracted
> Line: 2125  Objects:      368 detected /      161 sextracted
> Line: 2150  Objects:      379 detected /      162 sextracted
> Line: 2175  Objects:      381 detected /      166 sextracted
> Line: 2200  Objects:      392 detected /      169 sextracted
> Line: 2225  Objects:      394 detected /      172 sextracted
> Line: 2250  Objects:      403 detected /      175 sextracted
> Line: 2275  Objects:      408 detected /      178 sextracted
> Line: 2300  Objects:      409 detected /      180 sextracted
> Line: 2325  Objects:      410 detected /      181 sextracted
> Line: 2350  Objects:      410 detected /      184 sextracted
> Line: 2375  Objects:      415 detected /      187 sextracted
> Line: 2400  Objects:      417 detected /      192 sextracted
> Line: 2425  Objects:      418 detected /      196 sextracted
> Line: 2450  Objects:      424 detected /      198 sextracted
> Line: 2475  Objects:      428 detected /      203 sextracted
> Line: 2500  Objects:      432 detected /      204 sextracted
> Line: 2525  Objects:      435 detected /      205 sextracted
> Line: 2550  Objects:      440 detected /      207 sextracted
> Line: 2575  Objects:      452 detected /      208 sextracted
> Line: 2600  Objects:      460 detected /      215 sextracted
> Line: 2625  Objects:      462 detected /      218 sextracted
> Line: 2650  Objects:      465 detected /      220 sextracted
> Line: 2675  Objects:      468 detected /      222 sextracted
> Line: 2700  Objects:      471 detected /      225 sextracted
> Line: 2725  Objects:      477 detected /      229 sextracted
> Line: 2750  Objects:      481 detected /      231 sextracted
> Line: 2775  Objects:      484 detected /      236 sextracted
> Line: 2800  Objects:      484 detected /      237 sextracted
> Line: 2825  Objects:      487 detected /      240 sextracted
> Line: 2850  Objects:      493 detected /      243 sextracted
> Line: 2875  Objects:      495 detected /      245 sextracted
> Line: 2900  Objects:      497 detected /      248 sextracted
> Line: 2925  Objects:      504 detected /      255 sextracted
> Line: 2950  Objects:      510 detected /      258 sextracted
> Line: 2975  Objects:      511 detected /      265 sextracted
> Line: 3000  Objects:      511 detected /      268 sextracted
> Line: 3025  Objects:      511 detected /      270 sextracted
> Line: 3050  Objects:      512 detected /      270 sextracted
> Line: 3075  Objects:      517 detected /      270 sextracted
> Line: 3100  Objects:      518 detected /      275 sextracted
> Line: 3125  Objects:      521 detected /      285 sextracted
> Line: 3150  Objects:      526 detected /      295 sextracted
> Line: 3175  Objects:      532 detected /      295 sextracted
> Line: 3200  Objects:      533 detected /      296 sextracted
> Line: 3225  Objects:      535 detected /      297 sextracted
> Line: 3250  Objects:      538 detected /      301 sextracted
> Line: 3275  Objects:      540 detected /      303 sextracted
> Line: 3300  Objects:      540 detected /      304 sextracted
> Line: 3325  Objects:      541 detected /      305 sextracted
> Line: 3350  Objects:      545 detected /      306 sextracted
> Line: 3375  Objects:      547 detected /      309 sextracted
> Line: 3400  Objects:      548 detected /      312 sextracted
> Line: 3425  Objects:      550 detected /      312 sextracted
> Line: 3450  Objects:      556 detected /      315 sextracted
> Line: 3475  Objects:      557 detected /      316 sextracted
> Line: 3500  Objects:      566 detected /      321 sextracted
> Line: 3525  Objects:      568 detected /      324 sextracted
> Line: 3550  Objects:      571 detected /      332 sextracted
> Line: 3575  Objects:      587 detected /      338 sextracted
> Line: 3600  Objects:      594 detected /      342 sextracted
> Line: 3625  Objects:      600 detected /      348 sextracted
> Line: 3650  Objects:      605 detected /      350 sextracted
> Line: 3675  Objects:      607 detected /      355 sextracted
> Line: 3700  Objects:      609 detected /      357 sextracted
> Line: 3725  Objects:      613 detected /      364 sextracted
> Line: 3750  Objects:      615 detected /      368 sextracted
> Line: 3775  Objects:      616 detected /      370 sextracted
> Line: 3800  Objects:      619 detected /      370 sextracted
> Line: 3825  Objects:      640 detected /      384 sextracted
> Line: 3850  Objects:      642 detected /      389 sextracted
> Line: 3875  Objects:      646 detected /      392 sextracted
> Line: 3900  Objects:      648 detected /      395 sextracted
> Line: 3922  Objects:      650 detected /      400 sextracted
> Line: 3925  Objects:      650 detected /      401 sextracted
> Line: 3950  Objects:      656 detected /      406 sextracted
> Line: 3975  Objects:      657 detected /      409 sextracted
> Line: 4000  Objects:      660 detected /      409 sextracted
> Line: 4025  Objects:      665 detected /      410 sextracted
> Line: 4050  Objects:      672 detected /      410 sextracted
> Line: 4075  Objects:      681 detected /      415 sextracted
      Objects: detected 686      / sextracted 552             

> Closing files
> 
> All done (in 1.5 s: 2716.4 lines/s , 366.3 detections/s)

In [23]:

ds9.mtv(exposure, frame=frame, title="SExtractor Detections")
with ds9.Buffering():
    for line in open("calexp-0283453_10.cat").readlines():
        if line.startswith("#"):
            continue
        fields = line.split()
        x = float(fields[3])
        y = float(fields[4])
        ds9.dot("+", x, y, frame=frame)
frame += 1

In [24]:

%%bash --err error
/Users/becker/src/psfex/trunk.r206/bin/psfex calexp-0283453_10.ldac

In [25]:

from IPython.display import Image, display
i = Image(filename='fwhm_calexp-0283453_10.png')
display(i)

In [26]:

i = Image(filename='ellipticity_calexp-0283453_10.png')
display(i)

In [27]:

%%bash --err error
sex -PARAMETERS_NAME default.param -FILTER_NAME default.conv -CATALOG_TYPE ASCII_HEAD -CATALOG_NAME calexp-0283453_10.psf.cat "calexp-0283453_10.fits[1]" -PSF_NAME calexp-0283453_10.psf

In [28]:

ds9.mtv(exposure, frame=frame, title="SExtractor Detections Using Psf")
with ds9.Buffering():
    for line in open("calexp-0283453_10.psf.cat").readlines():
        if line.startswith("#"):
            continue
        fields = line.split()
        x = float(fields[3])
        y = float(fields[4])
        ds9.dot("+", x, y, frame=frame)
frame += 1