CSW is the OGC Catalogue Service It provides an interface for searching metadata catalogues. NIWA has its own metadata catalogue at http://dc.niwa.co.nz. We can use this to find some interesting data assets.
see http://geopython.github.io/OWSLib/#csw for more details
from owslib.csw import CatalogueServiceWeb
from owslib import fes
# connect to CSW, explore it's properties
#endpoint = 'http://www.ngdc.noaa.gov/geoportal/csw' # NGDC Geoportal
#endpoint = 'http://www.nodc.noaa.gov/geoportal/csw' # NODC Geoportal: granule level
#endpoint = 'http://data.nodc.noaa.gov/geoportal/csw' # NODC Geoportal: collection level
#endpoint = 'http://geodiscover.cgdi.ca/wes/serviceManagerCSW/csw' # NRCAN CUSTOM
#endpoint = 'http://geoport.whoi.edu/gi-cat/services/cswiso' # USGS Woods Hole GI_CAT
#endpoint = 'http://cida.usgs.gov/gdp/geonetwork/srv/en/csw' # USGS CIDA Geonetwork
#endpoint = 'http://cmgds.marine.usgs.gov/geonetwork/srv/en/csw' # USGS Coastal and Marine Program
endpoint = 'http://dc.niwa.co.nz/niwa_dc/srv/en/csw' # NIWA Metadata Catalogue
csw = CatalogueServiceWeb(endpoint)
print "Version", csw.version
print "Type", csw.identification.type
# note that setting username and password will allow more results to be returned if you are niwa admin.
print "username", csw.username
Version 2.0.2 Type CSW username None
# List the available CSW operations
[op.name for op in csw.operations]
['GetCapabilities', 'DescribeRecord', 'GetDomain', 'GetRecords', 'GetRecordById', 'Transaction', 'Harvest']
The getdomain call tells us what will be in the results record after the GetRecords Call.
csw.getdomain('GetRecords.resultType')
csw.results
{'parameter': 'GetRecords.resultType', 'type': 'csw:Record', 'values': ['hits', 'results', 'results_with_summary', 'validate']}
# setup a bounding box, this is the central north island.
NorthIsland=[172,-35, 179, -40]
# get records in this area matching the keyword fish:
csw.getrecords(keywords=['fish'],bbox=NorthIsland,maxrecords=20, resulttype="hits")
print "results", csw.results
results {'matches': 24, 'nextrecord': 21, 'returned': 20}
/Users/watkinsav/anaconda/lib/python2.7/site-packages/OWSLib-0.8.10-py2.7.egg/owslib/csw.py:196: UserWarning: Please use the updated 'getrecords2' method instead of 'getrecords'. The 'getrecords' method will be upgraded to use the 'getrecords2' parameters in a future version of OWSLib. in a future version of OWSLib.""")
The result csw.records is map with key = Metadata GUID and value holding the metadata content.
csw.getrecords(keywords=['fish'],bbox=NorthIsland,maxrecords=20, resulttype="results")
for rec in csw.records:
print rec + ":", csw.records[rec].title
4753f707-183e-4828-80fc-05d1d10a6b92: BOI Dropped Underwater Video (DUV) data in the Bay of Islands - RV Hawere b42896e9-c262-4881-b241-c15dc45e637f: BOI Demersal trawl data f7138677-f6a6-452b-b9a0-2b27a81998d7: Bay of Islands OS20/20 Survey Report Chapter 09: Seafloor assemblage and habitat assessment using DTIS 2518887e-28cc-48be-855f-9ea779be8daa: Bay of Islands OS20/20 Survey Report Chapter 13: Fish communities 1b3464c0-1598-44c6-90f0-f931b3c0cbcf: BOI Baited Underwater Video - shallow waters 43cd7eb8-9291-48de-ac0b-6c9db1720a9c: BOI Public presentations July 2010 - Ministry of Fisheries c431d79b-c729-40ee-b21e-2d32556d2439: Bay of Islands OS20/20 Survey Report Chapter 08: Water column and water quality f7bd7c6f-db48-4bc8-8f6f-152f8afc29d8: BOI Baited Underwater Video - deep waters bd903f15-0bf9-4c86-866b-2d59f2aa1861: Bay of Islands OS20/20 Survey Report Chapter 10: Shallow rocky reefs 4a6d04ef-8e28-08ca-e682-ad589b3c26f8: Massey Periphyton 2003 81219eba-4cc6-4ced-ae14-7f7f7b099695: NIWA Atlas of NZ Freshwater Fishes b96c15ae-061b-ac8f-8b37-931d3538f45e: Patea River periphyton e99e68fa-10f8-517f-8da9-bf9d3dfb3684: Biodiversity of springs and rock seepages 2003 d929d58f-dfa5-583a-75d1-ed34b3d450b0: Freshwater biodiversity 66dd6f24-e55e-4c04-9047-afac9778b155: Benthic 2000 Data 8d1ecc06-5d26-40aa-b6fc-74ed9bba49cb: Macrophyte Lake Data 90f1aadd-c2fa-4b2c-af83-5e2a5a98e89d: Cawthron Institute freshwater invertebrate and algae data 8cf44974-57b4-4ce7-8c7e-2843d1fe603e: National River Water Quality Network Database (Macro-invertebrates) df676dde-98ed-4141-b497-03efe0b0a24d: New Zealand Freshwater Fish Database 108c1900-a7d5-d3dd-b9cf-224e3ada41ed: Diversity of algae in wetlands
record = csw.records['90f1aadd-c2fa-4b2c-af83-5e2a5a98e89d']
record.title
record.xml
'<csw:SummaryRecord xmlns:dct="http://purl.org/dc/terms/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:geonet="http://www.fao.org/geonetwork" xmlns:csw="http://www.opengis.net/cat/csw/2.0.2" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">\n <dc:identifier>90f1aadd-c2fa-4b2c-af83-5e2a5a98e89d</dc:identifier>\n <dc:title>Cawthron Institute freshwater invertebrate and algae data</dc:title>\n <dc:type>dataset</dc:type>\n <dc:subject>invertebrate algae freshwater FBIS</dc:subject>\n <dc:subject>biota</dc:subject>\n <dc:format>csv</dc:format>\n <dct:abstract>Over the past 20+ years the Cawthron Institute has accumulated a large store of information on the nation-wide distribution of freshwater invertebrates and algae. This dataset covers 20 years worth of macoinvertebrate and algal samples processed by Cawthron for regional councils State of the Environment monitoring, funded research work, resource compliance monitoring surveys, and assessments of environmental effects.</dct:abstract>\n </csw:SummaryRecord>\n '
id = '81219eba-4cc6-4ced-ae14-7f7f7b099695'
csw.getrecordbyid(id=[id])
print csw.records[id].title
NIWA Atlas of NZ Freshwater Fishes
# FES is the Filter Encoding Standard.
from owslib.fes import PropertyIsEqualTo, PropertyIsLike, BBox
# Fish anywhere in the text
fish_query = PropertyIsEqualTo('csw:AnyText', 'fish')
csw.getrecords2(constraints=[fish_query], maxrecords=20)
print csw.results
# for rec in csw.records:
# print rec, csw.records[rec].title
# fish in the subject line
fish_query_like = PropertyIsLike('dc:subject', '%fish%')
csw.getrecords2(constraints=[fish_query_like], maxrecords=20)
print csw.results
{'matches': 40, 'nextrecord': 21, 'returned': 20} {'matches': 16, 'nextrecord': 0, 'returned': 16}
use getrecords2 for the constraints parameter
bbox_query = BBox([172,-35, 179, -40])
fish_query = PropertyIsEqualTo('csw:AnyText', 'fish')
csw.getrecords2(constraints=[fish_query, bbox_query], maxrecords=20)
print csw.results
# fish in the subject line
fish_query_like = PropertyIsLike('dc:subject', '%fish%')
csw.getrecords2(constraints=[fish_query_like, bbox_query], maxrecords=20)
print csw.results
{'matches': 21, 'nextrecord': 21, 'returned': 20} {'matches': 9, 'nextrecord': 0, 'returned': 9}
for rec in csw.records:
print rec, csw.records[rec].title
4753f707-183e-4828-80fc-05d1d10a6b92 BOI Dropped Underwater Video (DUV) data in the Bay of Islands - RV Hawere b42896e9-c262-4881-b241-c15dc45e637f BOI Demersal trawl data 2518887e-28cc-48be-855f-9ea779be8daa Bay of Islands OS20/20 Survey Report Chapter 13: Fish communities 1b3464c0-1598-44c6-90f0-f931b3c0cbcf BOI Baited Underwater Video - shallow waters f7bd7c6f-db48-4bc8-8f6f-152f8afc29d8 BOI Baited Underwater Video - deep waters bd903f15-0bf9-4c86-866b-2d59f2aa1861 Bay of Islands OS20/20 Survey Report Chapter 10: Shallow rocky reefs 81219eba-4cc6-4ced-ae14-7f7f7b099695 NIWA Atlas of NZ Freshwater Fishes df676dde-98ed-4141-b497-03efe0b0a24d New Zealand Freshwater Fish Database d565759a-9cf4-481e-be46-d742f6831222 BOI Public presentations July 2010 - Fish assemblages
wfs_query = PropertyIsEqualTo('csw:AnyText', 'wfs')
csw.getrecords2(constraints=[wfs_query], maxrecords=20)
from IPython.display import HTML, display
def print_dc(r):
display( HTML("<h3>%s <small>%s</small></h3>" % (r.title, r.identifiers[0]['identifier'])))
display( HTML("<p><strong>Subjects:</strong> %s</p>" % ", ".join(r.subjects)))
display( HTML("<p><strong>Abstract:</strong> %s</p>" % r.abstract))
display( HTML("<p><strong>Rights:</strong> %s</p>" % ", ".join(r.rights)))
# print "date:", r.date
# print "URIs:", A.uris
for u in r.uris:
display(HTML("<a href='%s'>%s</a>" % (u['url'], u['description'])))
print r.bbox
for rec in csw.records:
print_dc(csw.records[rec])
Subjects: field station data SIMS information management
Abstract: Archive for Metadata about NIWA's permanent field measurements
Rights:
None
Subjects: field station data, SIMS, information, management
Abstract: This is an OGC Web Feature Service feed from the NIWA Station management system, a component of the NIWA Spatial Data Infrastructure (SDI). It provides position and attribute information on over 5000 fixed position environmental monitoring stations across NZ and the Pacific. The service provides three feature sets: stations NIWA public stations -180°-+180° stations_360 NIWA public stations 0°360° The stations_360 layer returns station locations in the range 0..360° making it easier to display stations on either side of the International Date Line. stationtypes NIWA station types The stationtypes layer returns the various types of stations available: climate, rainfall, sea level etc. This allows clients to craft filtered requests or display different station types in different colours or styles.
Rights:
None
id = '81219eba-4cc6-4ced-ae14-7f7f7b099695'
csw.getrecordbyid(id=[id])
print_dc(csw.records[id])
Subjects: freshwater fish, fish species, geographic distribution, species descriptions
Abstract: This atlas contains a brief summary for each species, including the marine wanderers. Each summary includes: a picture of the fish, how to distinguish that fish from others, some interesting facts about the species and a distribution map.
Rights:
<owslib.ows.BoundingBox object at 0x106c95050>