from __future__ import print_function
from __future__ import division
import sys # following will be explained further down
import os.path
scriptpath = os.path.abspath('../scripts')
sys.path.append(scriptpath)
import conesearch
The following data structures have intrinsic syntactic support in Python (e.g., don't need explicit names to create
""
,''
,""""""
,''''''
[]
Sequence of objects indexed by number and mutable (size and entries can be changed)()
Sequence of objects indexed by number and immutable{}
Set of objects indexed by a keyDictionaries are a fundamental data structure in Python and extremely useful. So far you have seen examples of dictionary-like things without getting a specific introduction to dictionaries. If you don't end up using dictionaries a lot when using Python, you are likely doing something wrong (or many things wrong!).
Dictionaries are basically a way of looking up things by name or value. All sorts of things can be used as keys for the look up; the only requirement is that they be "immutable". Strings, tuples, numbers and such are all acceptable. Lists, arrays, and dictionaries are not since they are mutable.
traffic = {'red':'stop','green':'go','yellow':'speed-up'} # empty dicts are {}
print(traffic['red'])
stop
Alternate but equivalent ways to create dictionaries
traffic = dict(red='stop', green='go', yellow='speed-up') # if keys are legal var names
traffic = dict([('red','stop'),('green','go'),('yellow','speed-up')])
traffic = dict(zip(['red','green','yellow'],['stop','go','speed-up']))
print(traffic['magenta'])
--------------------------------------------------------------------------- KeyError Traceback (most recent call last) <ipython-input-4-3fcdffb9d3c3> in <module>() ----> 1 print(traffic['magenta']) KeyError: 'magenta'
ERROR: KeyError: 'magenta' [IPython.core.interactiveshell]
# can add new dictionary items
traffic['broken'] = 'crash'
traffic[5] = 'the low spark of high heeled boys'
print(traffic)
{'broken': 'crash', 'green': 'go', 'yellow': 'speed-up', 'red': 'stop', 5: 'the low spark of high heeled boys'}
Many dictionary methods are available
print(traffic.keys()) # note keys have no guaranteed order.
print(traffic.values())
print(traffic.items())
['broken', 'green', 'yellow', 'red', 5] ['crash', 'go', 'speed-up', 'stop', 'the low spark of high heeled boys'] [('broken', 'crash'), ('green', 'go'), ('yellow', 'speed-up'), ('red', 'stop'), (5, 'the low spark of high heeled boys')]
print('red' in traffic)
print('magenta' in traffic)
True False
# getting a default value if it doesn't have a key
traffic.get('blue','fasten your seatbelt and drive carefully')
'fasten your seatbelt and drive carefully'
traffic[[1,2]] = '?'
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-9-987630994cde> in <module>() ----> 1 traffic[[1,2]] = '?' TypeError: unhashable type: 'list'
ERROR: TypeError: unhashable type: 'list' [IPython.core.interactiveshell]
# tuples work though
traffic[(1,2)] = '?'
print(traffic[(1,2)])
?
Note that much of the pyfits header and astropy table interfaces behave much like dictionaries though they are not actually dictionaries.
Storing information based on names or modes. For example:
One can use anything as a dictionary value, including functions. This is a useful way of directing flow by a key value
def fromfits(filename):
"""read spectrum from fits file"""
...
return wavelength, flux
def fromascii(filename):
"""read spectrum from a text file"""
...
return wavelength, flux
def fromhdf(filename):
"""read spectrum from hdf5"""
...
return wavelength, flux
spectrumio_switch = {'fits':fromfits, 'dat':fromascii, 'hd5':fromhdf}
def get_spec_data(filename):
"""return wavelength and flux arrays for a variety of supported formats"""
return spectrumio_switch[os.path.splitext(filename)[1]](filename)
Note that this allows dynamic changes to this function. If I give this module to others, they can define their own special handler for a format not supported in mine, add it to the module dictionary, and now all code that uses get_spec_data() automatically uses their option without changing a line of code in the original module.
The PIL (Python Imaging Library) is capable of dealing with the reading and writing of many image formats (it is likely to be replaced by pillow, with most of the same interface). It is simple to convert to and from numpy arrays.
Supported formats include:
BMP
BUFR (identify only)
CUR (read only)
DCX (read only)
EPS (write-only)
FITS (identify only)
FLI, FLC (read only)
FPX (read only)
GBR (read only)
GD (read only)
GIF
GRIB (identify only)
HDF5 (identify only)
ICO (read only)
IM
IMT (read only)
IPTC/NAA (read only)
JPEG
MCIDAS (read only)
MIC (read only)
MPEG (identify only)
MSP
PALM (write only)
PCD (read only)
PCX
PDF (write only)
PIXAR (read only)
PNG
PPM
PSD (read only)
SGI (read only)
SPIDER
TGA (read only)
TIFF
WAL (read only)
WMF (identify only)
XBM
XPM (read only)
XV Thumbnails
import Image
im = Image.open('../data/saturn.jpg')
nim = np.array(im) # numpy knows how to convert
print(nim.shape) # shows 3d array, last dim are the rgb planes
imshow(nim)
(263, 350, 3)
<matplotlib.image.AxesImage at 0x108382d90>
im.save('saturn.tiff')
im2 = Image.fromarray((nim.max()-nim)[::2])
im2.save('saturn2.tiff')
Documentation on all the available functionality of PIL is here: http://www.pythonware.com/library/pil/handbook/index.htm
This is a short introduction to handling Exceptions in Python.
Further reading: http://docs.python.org/2/tutorial/errors.html
Rather than require all code to check for errors for any function called (this leads to very cumbersome and hard to maintain code), Python, like many modern languages, uses exceptions to deal with errors.
Typically, when a problem is encountered in Python code, the general pattern is to "raise" an exception. If this happens in a function or code block, processing stops, and the exception is passed along to the code that called the function. If there isn't any special code there to deal with that specific problem, it continues to get passed to higher levels of code. If nothing deals with it, execution stops, and an error traceback is printed showing all the levels involved, where the execution in each level was at, and what exception was raised. In this way, you don't need to write code to deal with exceptions if you don't expect them, or know how to deal with them. But you can if you do. Sometimes code is written that uses them to deal with common issues that may arise by trying normal case, and in the case of a problem, trying an alternative.
Suppose we are trying to convert values in a table into floating point numbers, but there may be some instances where the table value isn't a floating point number, but you don't want to stop processing because of that, but instead use a default value, like a NaN value
float("N/A") # raises an exeption since it can't be converted to float
--------------------------------------------------------------------------- ValueError Traceback (most recent call last) <ipython-input-13-c8aef96cbe70> in <module>() ----> 1 float("N/A") # raises an exeption since it can't be converted to float ValueError: could not convert string to float: N/A
ERROR: ValueError: could not convert string to float: N/A [IPython.core.interactiveshell]
The standard way to deal with these is to use "try except" blocks
def robust_float(fstring):
try:
x = float(fstring)
except ValueError as e:
# we can find out more detail about the exception
print(e.args)
print(e.message)
x = np.nan
return x
print(robust_float("3.14"))
print(robust_float("N/A"))
3.14 ('could not convert string to float: N/A',) could not convert string to float: N/A nan
It is ESSENTIAL practice to only catch the specific types of Exceptions you are prepared to deal with. If you trap all exceptions (i.e., "except Exception") you run the risk of an unexpected problem being ignored, and you won't understand where the problem occurred since you explicitly ignored it. This can be extremely hard to track down.
def robust_inverted_float(fstring):
try:
x = 1./float(fstring)
# To catch multiple types of Exceptions:
# except (ThisError, ThatError) as ...
except ValueError:
print("Warning, encountered non-float value: "+fstring)
x = np.nan
except ZeroDivisionError:
print("Warning, encountered divide by zero")
x = np.nan
return x
print(robust_inverted_float("3.3"))
print(robust_inverted_float("N/A"))
print(robust_inverted_float("0"))
0.30303030303 Warning, encountered non-float value: N/A nan Warning, encountered divide by zero nan
You can also use else and finally in the logic. The else block is executed if no exception happened. The finally block always is executed regardless of whether exceptions occur. This is handy if you need to close a file or delete a resource regardless (though the "with" block handles many of these cases now).
ImportError is useful when you want to disable something in your code if a dependency is not found.
try:
import myplottingpackage
doplot = True
except ImportError:
print('The required plotting package is not found')
doplot = False
# some codes to get results, and then...
print('Doing science...')
result = 42
if doplot:
myplottingpackage.fancyplot(result)
else:
print('No plotting done.')
The required plotting package is not found Doing science... No plotting done.
You can also define your own type of Exception. This is mostly useful so that you can catch these specfic errors else where in code. While this uses classes, you don't need to know much more than blindly following this example. Note that your definition of an exception will also be considered a match for each exception in its inheritance hierarchy (this part does require a little understanding of classes: (http://docs.python.org/2/tutorial/classes.html).
# This inherits everything from Exception class
class PhotError(Exception):
"""Photometry errors."""
pass
raise PhotError('My phot failed. Oh, no!')
--------------------------------------------------------------------------- PhotError Traceback (most recent call last) <ipython-input-18-c605b2a32222> in <module>() ----> 1 raise PhotError('My phot failed. Oh, no!') PhotError: My phot failed. Oh, no!
ERROR: PhotError: My phot failed. Oh, no! [IPython.core.interactiveshell]
In terms of code performance when handling Exceptions, this discussion might be useful: http://stackoverflow.com/questions/5589532/try-catch-or-validation-for-speed
Basically one should avoid using exceptions for the usual case since it is slower than testing for conditions.
One can use the os, os.path, sys, glob, and shutil modules to do many of the interactions with files that one would normally do from the shell. The following shows examples of common things, but refer to the library documentation on these for all the available functions:
# sys provides information about the current session. Most useful for most people
# is the current path information, version of python, and the ability to manipulate
# stdin, stdout, stderr
import sys
print(sys.path)
print(sys.version)
oldstdout = sys.stdout # save it!
sys.stdout = open('stdout.txt','w')
print("hello world")
sys.stdout.close()
sys.stdout = oldstdout
['', '/Users/perry/Ureka/variants/common/lib/python', '/Users/perry/Ureka/variants/common/lib/python', '/Users/perry/Ureka/python/lib', '/Users/perry/Ureka/python/lib/python2.7/site-packages/distribute-0.6.28-py2.7.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_core-2.3-py2.7-macosx-10.6-x86_64.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc-2.3-py2.7.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_ServiceManagement-2.3-py2.7.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_ServerNotification-2.3-py2.7.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_CoreLocation-2.3-py2.7.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_AppleScriptObjC-2.3-py2.7.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_ScriptingBridge-2.3-py2.7.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_QTKit-2.3-py2.7.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_PubSub-2.3-py2.7.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_InterfaceBuilderKit-2.3-py2.7.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_InstantMessage-2.3-py2.7.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_InputMethodKit-2.3-py2.7.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_DictionaryServices-2.3-py2.7.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_Collaboration-2.3-py2.7.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_CalendarStore-2.3-py2.7.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_XgridFoundation-2.3-py2.7.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_WebKit-2.3-py2.7.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_SystemConfiguration-2.3-py2.7-macosx-10.6-x86_64.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_SyncServices-2.3-py2.7.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_SearchKit-2.3-py2.7.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_ScreenSaver-2.3-py2.7-macosx-10.6-x86_64.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_Quartz-2.3-py2.7-macosx-10.6-x86_64.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_PreferencePanes-2.3-py2.7.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_Message-2.3-py2.7.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_LaunchServices-2.3-py2.7.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_LatentSemanticMapping-2.3-py2.7.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_InstallerPlugins-2.3-py2.7.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_FSEvents-2.3-py2.7-macosx-10.6-x86_64.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_ExceptionHandling-2.3-py2.7.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_CoreText-2.3-py2.7-macosx-10.6-x86_64.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_CoreData-2.3-py2.7.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_Cocoa-2.3-py2.7-macosx-10.6-x86_64.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_CFNetwork-2.3-py2.7-macosx-10.6-x86_64.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_Automator-2.3-py2.7.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_AppleScriptKit-2.3-py2.7.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pyobjc_framework_AddressBook-2.3-py2.7.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/tornado-2.3-py2.7.egg', '/Users/perry/Ureka/python/lib/python2.7/site-packages/pip-1.1-py2.7.egg', '/Users/perry/Ureka/python/lib/python27.zip', '/Users/perry/Ureka/python/lib/python2.7', '/Users/perry/Ureka/python/lib/python2.7/plat-darwin', '/Users/perry/Ureka/python/lib/python2.7/plat-mac', '/Users/perry/Ureka/python/lib/python2.7/plat-mac/lib-scriptpackages', '/Users/perry/Ureka/python/lib/python2.7/lib-tk', '/Users/perry/Ureka/python/lib/python2.7/lib-old', '/Users/perry/Ureka/python/lib/python2.7/lib-dynload', '/Users/perry/Ureka/python/lib/python2.7/site-packages', '/Users/perry/Ureka/python/lib/python2.7/site-packages/PIL', '/Users/perry/Ureka/python/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg-info', '/Users/perry/Ureka/variants/common/lib/python', '/Users/perry/Ureka/variants/common/lib/python', '/Users/perry/Ureka/python/lib', '/Users/perry/Ureka/python/lib/python2.7/site-packages/IPython/extensions', '/Users/perry/Dropbox/gitfork/scientific-python-training-2012/scripts'] 2.7.3 (default, Nov 6 2012, 09:20:07) [GCC 4.2.1 (Apple Inc. build 5664)]
!head stdout.txt
hello world
# glob is useful for getting a list of files similar to that you would have through
# the shell ls command with wildcards
import glob
flist = glob.glob('*.ipynb')
print(flist)
['PSF_Photometry_Process.ipynb', 'Session1_Astro_Demo.ipynb', 'Session1_Demo.ipynb', 'Session1_Introduction.ipynb', 'Session2_Basic.ipynb', 'Session3_Phot_Phun_1.ipynb', 'Session4_phot_2.ipynb', 'Session5_STIS_Spec.ipynb', 'Session6_Exceptions.ipynb', 'Session6_Various_Topics.ipynb']
# os.path is useful for defining file paths that do not depend on os variations (particularly
# for MS windows), and information about particular files, and lots of useful path parsing
# utilities
import os.path
path = os.path.join('$HOME', 'bin')
print(path)
fpath = 'Session1_Introduction.ipynb'
print(os.path.exists(fpath))
apath = os.path.abspath(fpath)
print("absolute path=",apath)
print("directory=",os.path.dirname(apath))
print("basename=",os.path.basename(apath))
$HOME/bin True absolute path= /Users/perry/Dropbox/gitfork/scientific-python-training-2012/lecture_notebooks/Session1_Introduction.ipynb directory= /Users/perry/Dropbox/gitfork/scientific-python-training-2012/lecture_notebooks basename= Session1_Introduction.ipynb
# shutil has several operations for manipulating files.
import shutil
shutil.copyfile(fpath, "temp.ipynb") # copy a file
print(glob.glob('*.ipynb'))
shutil.move("temp.ipynb", "..")
print(glob.glob('../*ipynb'))
['PSF_Photometry_Process.ipynb', 'Session1_Astro_Demo.ipynb', 'Session1_Demo.ipynb', 'Session1_Introduction.ipynb', 'Session2_Basic.ipynb', 'Session3_Phot_Phun_1.ipynb', 'Session4_phot_2.ipynb', 'Session5_STIS_Spec.ipynb', 'Session6_Exceptions.ipynb', 'Session6_Various_Topics.ipynb', 'temp.ipynb'] ['../temp.ipynb']
# os has many OS level functions
import os
print(os.environ) # list all environmental variables
os.chdir('..')
print("\n\nCurrent working directory = ", os.getcwd())
print(os.listdir('.'))
os.mkdir('bozo')
print(os.listdir('.'))
os.rmdir('bozo')
os.remove('temp.ipynb')
# and many more functions to deal with processes, file permissions, etc.
{'hbin': '/Users/perry/Ureka/iraf//unix/bin.macosx/', 'HSI_LF': '-I/Users/perry/Ureka/iraf//include/ -arch i386 -m32 -mmacosx-version-min=10.4', 'GROUP': 'STSCI\\science', 'XDG_DATA_HOME': '/Users/perry/.local/share', 'REMOTEHOST': '', 'TERM_PROGRAM_VERSION': '303.2', 'UR_DIR': '/Users/perry/Ureka', 'LOGNAME': 'perry', 'USER': 'perry', 'PATH': '/Users/perry/Ureka/bin:/Users/perry/Ureka/variants/common/bin:/Users/perry/Ureka/variants/common/bin:/Users/perry/Ureka/python:/Users/perry/Ureka/python/bin:/Users/perry/bin:/Applications/ST_sci/SM/bin:/Applications/ST_sci/SEXTRACTOR/bin:/sw/bin:/sw/sbin:/Applications/ST_sci/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin:/usr/texbin:/usr/X11R6/bin', 'HOME': '/Users/perry', 'CLICOLOR': '1', 'UR_DIR_PKG': '/Users/perry/Ureka/variants/common/', 'DISPLAY': '/tmp/launch-uG4eFP/org.x:0', 'UR_OS': 'osx-7', 'TERM_PROGRAM': 'Apple_Terminal', 'LANG': 'en_US.UTF-8', 'INFOPATH': '/sw/share/info:/sw/info:/usr/share/info', 'TERM': 'xterm-color', 'SHELL': '/bin/tcsh', 'HSI_LIBS': '/Users/perry/Ureka/iraf//unix/hlib/libboot.a /Users/perry/Ureka/iraf//lib/libsys.a /Users/perry/Ureka/iraf//lib/libvops.a /Users/perry/Ureka/iraf//unix/hlib/libos.a ', 'SHLVL': '1', 'GDFONTPATH': '/Library/Fonts', 'HEADAS': '/Applications/ST_sci/heasoft-6.11.1/i386-apple-darwin11.1.0', 'EXELIS_DIR': '/Applications/exelis', 'PGPLOT_DIR': '/sw/lib/pgplot', 'GNUTERM': 'aqua', 'GNUPLOT_DEFAULT_GDFONT': 'Arial', 'MANPATH': '/Users/perry/Ureka/share/man:/Users/perry/Ureka/variants/common/man:/Users/perry/Ureka/variants/common/man:/Users/perry/Ureka/variants/common/share/man:/Users/perry/Ureka/python/man::/Applications/ST_sci/SEXTRACTOR/man:/sw/share/man:/usr/share/man:/usr/local/share/man:/usr/X11/man:/usr/texbin/man:/sw/lib/perl5/5.12.3/man:/usr/X11R6/man', 'OSTYPE': 'darwin', 'tmp': '/tmp/', 'HSI_LFLAGS': '', 'UR_CPU': 'x86', 'XDG_CONFIG_HOME': '/Users/perry/.config', 'TERM_SESSION_ID': 'B8884652-91FC-4314-AFF6-5CF2E6E32C55', 'XAPPLRESDIR': '/sw/etc/app-defaults/', 'VENDOR': 'apple', 'ac_cv_path_INTLTOOL_PERL': '/usr/bin/perl', 'GIT_PAGER': 'cat', 'HSI_XF': '-I/Users/perry/Ureka/iraf//include/ -Inolibc -/DMACOSX -w -/Wunused -/m32 -/arch -//i386 -/mmacosx-version-min=10.4', 'CC': 'cc', 'iraf': '/Users/perry/Ureka/iraf/', 'SSH_AUTH_SOCK': '/tmp/launch-vJbLQ1/Listeners', 'XDG_CACHE_HOME': '/Users/perry/.cache', 'HSI_CF': '-I/Users/perry/Ureka/iraf//include/ -O -DMACOSX -w -Wunused -arch i386 -m32 -mmacosx-version-min=10.4', 'XML_CATALOG_FILES': '/sw/etc/xml/catalog', 'DYLD_LIBRARY_PATH': '/Applications/ST_sci/SM/lib:', 'HOST': 'xyzzy.stsci.edu', 'Apple_PubSub_Socket_Render': '/tmp/launch-nTswDn/Render', 'HOSTTYPE': 'intel-mac', 'XDG_DATA_DIRS': '/sw/share', 'IRAFARCH': 'macosx', 'HSI_FF': '-I/Users/perry/Ureka/iraf//include/ -O -arch i386 -m32 -DBLD_KERNEL -mmacosx-version-min=10.4', 'IDL_DIR': '/Applications/exelis/idl82', 'F77': '/Users/perry/Ureka/iraf//unix/hlib//f77.sh', 'RANLIB': 'ranlib', 'DBUS_SESSION_BUS_ADDRESS': 'launchd:env=DBUS_FINK_SESSION_BUS_SOCKET', 'TMPDIR': '/var/folders/m1/8vfhq2315vlcqkbn6bhh4wgh0003y2/T/', 'MACH': 'macosx', 'F2C': '/Users/perry/Ureka/iraf//unix/bin.macosx//f2c.e', 'PERL5LIB': '/sw/lib/perl5:/sw/lib/perl5/darwin', 'hostid': 'unix', 'XDG_CONFIG_DIRS': '/sw/etc/xdg', 'HSI_F77LIBS': '', 'UR_PYTHONPATH': '/Users/perry/Ureka/variants/common/lib/python:/Users/perry/Ureka/variants/common/lib/python:/Users/perry/Ureka/python/lib', 'UR_BITS': '64', 'GDK_USE_XFT': '1', 'IDL_PATH': '.:+/Applications/exelis/local/astron:+/Applications/exelis/idl82/lib', 'HSI_OSLIBS': '', 'host': '/Users/perry/Ureka/iraf//unix/', 'hlib': '/Users/perry/Ureka/iraf//unix/hlib/', '__CF_USER_TEXT_ENCODING': '0xFC2:0:0', 'Apple_Ubiquity_Message': '/tmp/launch-zaIe5b/Apple_Ubiquity_Message', 'PWD': '/Users/perry/dropbox/gitfork/scientific-python-training-2012/lecture_notebooks', 'SGML_CATALOG_FILES': '/sw/etc/sgml/catalog', 'PAGER': 'cat', 'GDL_PATH': '+/sw/share/gnudatalanguage/lib', 'UR_VARIANT': 'common', 'MACHTYPE': 'x86_64', 'QTDIR': '/sw/lib/qt3', 'COMMAND_MODE': 'unix2003'} Current working directory = /Users/perry/Dropbox/gitfork/scientific-python-training-2012 ['.git', '.gitignore', 'data', 'homework_notebooks', 'lecture_notebooks', 'README.rst', 'scripts', 'temp.ipynb'] ['.git', '.gitignore', 'bozo', 'data', 'homework_notebooks', 'lecture_notebooks', 'README.rst', 'scripts', 'temp.ipynb']
With the supplied conesearch.py module, it is fairly simple to read conesearch results into python. The main trick now is finding the appropriate URL to use for the catalog you wish to use for a conesearch (In astropy 0.3 an easier-to-use and more robust conesearch facility will be available). We'll show an example of obtaining a list of astrometric starts from the USNO-A2 catalog that are within 0.1 degree radius of ra=10, dec=10 (both in degrees).
The following link has more info on this particular conesearch and the URL's of related catalogs http://www.nofs.navy.mil/data/fchpix/vo_nofs.html
results = conesearch.vo_service_request(
"http://www.nofs.navy.mil/cgi-bin/vo_cone.cgi?CAT=USNO-B1&",
RA=10.,DEC=10.,SR=0.1)
Downloading http://www.nofs.navy.mil/cgi-bin/vo_cone.cgi?CAT=USNO-B1&SR=0.1&DEC=10.0&RA=10.0 [Done] WARNING: W21: http://www.nofs.navy.mil/cgi-bin/vo_cone.cgi?CAT=USNO-B1&SR=0.1&DEC=10.0&RA=10.0:4:0: W21: vo.table is designed for VOTable version 1.1 and 1.2, but this file is 1.0 [astropy.io.votable.exceptions] WARNING: W42: http://www.nofs.navy.mil/cgi-bin/vo_cone.cgi?CAT=USNO-B1&SR=0.1&DEC=10.0&RA=10.0:4:0: W42: No XML namespace specified [astropy.io.votable.exceptions] WARNING: W17: http://www.nofs.navy.mil/cgi-bin/vo_cone.cgi?CAT=USNO-B1&SR=0.1&DEC=10.0&RA=10.0:15:15: W17: VOTABLE element contains more than one DESCRIPTION element [astropy.io.votable.exceptions]
help(results)
Help on Table in module astropy.io.votable.tree object: class Table(Element, _IDProperty, _NameProperty, _UcdProperty, _DescriptionProperty) | TABLE_ element: optionally contains data. | | It contains the following publicly-accessible and mutable | attribute: | | *array*: A Numpy masked array of the data itself, where each | row is a row of votable data, and columns are named and typed | based on the <FIELD> elements of the table. The mask is | parallel to the data array, except for variable-length fields. | For those fields, the numpy array's column type is "object" | (``"O"``), and another masked array is stored there. | | If the Table contains no data, (for example, its enclosing | :class:`Resource` has :attr:`~Resource.type` == 'meta') *array* | will have zero-length. | | The keyword arguments correspond to setting members of the same | name, documented below. | | Method resolution order: | Table | Element | _IDProperty | _NameProperty | _UcdProperty | _DescriptionProperty | __builtin__.object | | Methods defined here: | | __init__(self, votable, ID=None, name=None, ref=None, ucd=None, utype=None, nrows=None, id=None, config={}, pos=None, **extra) | | create_arrays(self, nrows=0, config={}) | Create a new array to hold the data based on the current set | of fields, and store them in the *array* and member variable. | Any data in the existing array will be lost. | | *nrows*, if provided, is the number of rows to allocate. | | get_field_by_id = lookup_by_id(self, ref, before=None) | Looks up a FIELD or PARAM element by the given ID. | | get_field_by_id_or_name = lookup_by_id(self, ref, before=None) | Looks up a FIELD or PARAM element by the given ID or name. | | get_group_by_id = lookup_by_id(self, ref, before=None) | Looks up a GROUP element by the given ID. Used by the group's | "ref" attribute | | is_empty(self) | Returns True if this table doesn't contain any real data | because it was skipped over by the parser (through use of the | `table_number` kwarg). | | iter_fields_and_params(self) | Recursively iterate over all FIELD and PARAM elements in the | TABLE. | | iter_groups(self) | Recursively iterate over all GROUP elements in the TABLE. | | parse(self, iterator, config) | | to_table(self) | Convert this VO Table to an `astropy.table.Table` instance. | | .. warning:: | | Variable-length array fields may not be restored | identically when round-tripping through the | `astropy.table.Table` instance. | | to_xml(self, w, **kwargs) | | ---------------------------------------------------------------------- | Class methods defined here: | | from_table(cls, votable, table) from __builtin__.type | Create a `Table` instance from a given `astropy.table.Table` | instance. | | ---------------------------------------------------------------------- | Data descriptors defined here: | | fields | A list of :class:`Field` objects describing the types of each | of the data columns. | | format | [*required*] The serialization format of the table. Must be | one of: | | 'tabledata' (TABLEDATA_), 'binary' (BINARY_), 'fits' (FITS_). | | Note that the 'fits' format, since it requires an external | file, can not be written out. Any file read in with 'fits' | format will be read out, by default, in 'tabledata' format. | | groups | A list of :class:`Group` objects describing how the columns | and parameters are grouped. Currently this information is | only kept around for round-tripping and informational | purposes. | | infos | A list of :class:`Info` objects for the table. Allows for | post-operational diagnostics. | | links | A list of :class:`Link` objects (pointers to other documents | or servers through a URI) for the table. | | nrows | [*immutable*] The number of rows in the table, as specified in | the XML file. | | params | A list of parameters (constant-valued columns) for the | table. Must contain only :class:`Param` objects. | | ref | | ---------------------------------------------------------------------- | Data descriptors inherited from Element: | | __dict__ | dictionary for instance variables (if defined) | | __weakref__ | list of weak references to the object (if defined) | | ---------------------------------------------------------------------- | Data descriptors inherited from _IDProperty: | | ID | The XML ID_ of the element. May be ``None`` or a string | conforming to XML ID_ syntax. | | ---------------------------------------------------------------------- | Data descriptors inherited from _NameProperty: | | name | An optional name for the element. | | ---------------------------------------------------------------------- | Data descriptors inherited from _UcdProperty: | | ucd | The `unified content descriptor`_ for the element. | | ---------------------------------------------------------------------- | Data descriptors inherited from _DescriptionProperty: | | description | An optional string describing the element. Corresponds to the | DESCRIPTION_ element.
for field in results.fields: print(field.name)
id RA DEC sRa sDec sRaEp sDeEp MuRA MuDEC sMuRA sMuDE B1 R1 B2 R2 I2
plot(results.array['RA'],results.array['DEC'],'+')
[<matplotlib.lines.Line2D at 0x108684d90>]
Regular expression tools provide extremely powerful tools for processing strings. But it is a complex topic that whole books have been devoted to. One should keep in mind the following quote:
'Some people, when confronted with a problem, think “I know, I'll use regular expressions.” Now they have two problems.'
Nevvertheless, they can be very useful; just try to keep particular regular expressions from getting too complicated. They can be hard to debug when complex, besides being hard to read.
The basic capabilities include pattern matching tools that allow for:
Basic reference is Python's library reference on the relevant module: http://docs.python.org/2/library/re.html
import re
s2 = "this line contains some decimal numbers such as 7, 9.55, and 3. for test purposes"
re_decimal = re.compile(r"\d*[.]\d+|\d+[.]?")
mo = re_decimal.findall(s2)
print(mo)
['7', '9.55', '3.']
s1 = "the email address is bozo@clowninstitute.com, and one other is rednose@clownsupply.com "
# if matching is to be repeated a lot, precompiling the expression speeds up execution
re_email = re.compile(r"\s\S+\@\S+[.]\S+\s") # note use of "raw" strings to prevent many \\
mo = re_email.search(s1)
print(mo.group())
mo = re_email.findall(s1)
print(mo)
bozo@clowninstitute.com, [' bozo@clowninstitute.com, ', ' rednose@clownsupply.com ']
This regular expression is not precise since it includes the whitespace and folds in surrounding punctuation. A really precise regular expression can be quite complex. Here is an example for email addresses that is more precise that was obtained from the internet (these can vary depending on what they restrict, or what future cases they permit)
r"[a-z0-9!#$%&'*+/=?^_
{|}~-]+(?:.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)@(?:[a-z0-9](?:[a-z0-9-][a-z0-9])?.)+(?:[A-Z]{2}|com|org|net|edu|gov|mil|biz|info|mobi|name|aero|asia|jobs|museum)\b"`
Rather than go to these lengths, it usually makes more sense to go with simpler matches and apply further filtering with successive python string processing code (e.g., stripping whitespace after the fact), or successive regular expressions.
Finally it is very important to note the distinction between the match()
and search()
methods. Match means the match must start at the beginning of the string, search tries to find it anywhere within the string, which is often what people want.
The standard debugger is pdb, but it doesn't yet work in the notebook. Presumably it will in future versions. It does work in the terminal version of ipython, which is where it will be demoed. The standard documentation on pdb is located at http://docs.python.org/2/library/pdb.html
In ipython, if one uses the %pdb magic, then if your program encounters an untrapped exception, ipython automatically invokes the debugger at the point of the exception, from which one can examine the state of all variables.
Python has a special object frequently used to indicate that no value is present called None
. Note that it shows differently depending on whether you print it or use the repr() mechanism (what ipython uses if you type it at the prompt without a print statement/function
x = None
print(x)
None
x
The best way to test for None is to use the following form:
print(x is None)
print(x is not None)
True False
Using an equality test (x == None) is not recommended since it can be confused with other values since some objects will consider a None as equal to them even if they are not None objects.
We have been using Python 2 for this course. The is a Python 3 that has been available for a few years that is incompatable with Python 2. Eventually we expect most of Python 2 development to switch to Python 3, but that isn't likely to happen for a year or two. Nevertheless, it makes sense to write existing Python 2 scripts and modules in a way that will minimize the transition pain. The two most useful things you can do is always put:
from __future__ import future_print
from __future__ import division
At the top of all scripts. The first forces you to use the python print function instead of the print statement. Python 3 only supports the print function.
The second makes the division behavior compatable with that in Python 3. In Python 2 the following is the usual behavior:
>>> 1/2
0
In Python 3, this is turned into a float (for all integer division cases regardless of whether the result is an integer or not):
>>> 1/2
0.5
This is the most important case of future proofing your code since finding cases where the difference in behavior is important to your code can be quite difficult.
There are number of differences between the two that are too extensive to go into here. Most of the advantages of Python 3 over 2 are not generally of great interest to scientists (e.g., much better support for unicode strings), and in some cases, more hassle (e.g., converting byte data into arrays or character strings). But the Python world doesn't want a permanant 2/3 split so we expect that eventually everything will migrate to 3. There is a 2to3 tool that helps translate v2 code into v3 code, but it doesn't handle everything (particularly integer division). There are ways of writing code that does convert perfectly, but understanding how to do that gets moderately involved, and most shouldn't worry obsessively about that. If you do the above, you will remove most of the headaches of the transition. The other issues generally will reveal themselves in obvious ways when it comes to it.
1/1
1.0