Open Context: Data Publication for Cultural Heritage and Field Research: "Open Context reviews, edits, and publishes archaeological research data and archives data with university-backed repositories, including the California Digital Library."
I often think of OpenContext as an examplar – a model from the future: academic data archiving done right. Some cool features (About Open Context: Technologies):
We want to provide for the long-term citability and availability of this data.
Also contextualization.
What was the excellent presentation/paper he made to us in WwOD13?
There are so many possibilities here; we can work iteratively with Eric Kansa to develop a good project without having it all figured out upfront.
Eric has mentioned to me the idea of time span facets.
How to reproduce data represented by the map on Open Context?
How to use the API to get a list of projects?
http://opencontext.org/sets/.json returns json representation of items, but http://opencontext.org/projects/.json doesn't work for getting list of all projects. Answer:
Let's use a specific project to focus on:
The API documentation: http://opencontext.org/about/services
# using an example in the API documentation to confirm that we can get json representation from API
import requests
json_url = "http://opencontext.org/sets/Palestinian+Authority/Tell+en-Nasbeh/.json?proj=Bade+Museum"
r = requests.get(json_url)
# what are the top level keys of response?
r.json().keys()
[u'updated', u'sorting', u'numFound', u'facets', u'offset', u'geoCount', u'chronoTileFacets', u'summary', u'paging', u'qstring', u'published', u'results', u'paramCount', u'geoTileFacets']
# Now let's apply same logic to the Asian Stoneware Jars project
json_url = "http://opencontext.org/sets/.json?proj=Asian+Stoneware+Jars"
request = requests.get(json_url)
request_json = request.json()
results= request_json['results']
request_json.keys()
[u'updated', u'sorting', u'numFound', u'facets', u'offset', u'geoCount', u'chronoTileFacets', u'summary', u'paging', u'qstring', u'published', u'results', u'paramCount', u'geoTileFacets']
# number of results matches what is on human UI
request_json['numFound']
1008
# we get back the first page of 10
len(results)
10
results[0]
{u'catIcon': u'http://opencontext.org/database/ui_images/med_oc_icons/ceramic_artifacts_50x50.jpg', u'category': u'Pottery', u'context': u'<div class="context"><div>\nContext: <span class="item_root_parent">Philippines</span> / <span class="item_parent">San Diego</span></div>\n</div>', u'geoTime': {u'geoLat': 13.539201, u'geoLong': 121.168213, u'timeBegin': False, u'timeEnd': False}, u'label': u'UNE 104', u'project': u'Asian Stoneware Jars', u'thumbIcon': u'http://artiraq.org/static/opencontext/stoneware-media/thumbs/photographs/Edited%20Copies/Copy%20(2)%20of%20une104%20copy.jpg', u'uri': u'http://opencontext.org/subjects/EAFD5A63-83C0-43A3-691C-08878757A66D', u'var_vals': {u'Artefact Type': u'intact jar', u'Compositional Group': u'1', u'Dataset Type': u'A Ship', u'Diameter (mm)': u'440', u'Donor Institution/sample Source': u'National Museum of the Philippines', u'Group (INAA)': u'1', u'Group (icp)': u'1', u'Height (mm)': u'530', u'ICP - Ba': u'475.54', u'ICP - Ca': u'1366.57', u'ICP - Ce': u'120.03', u'ICP - Cu': u'27.82', u'ICP - Fe': u'14855.69', u'ICP - Ga': u'27.29', u'ICP - Hf': u'4.82', u'ICP - K': u'21721.65', u'ICP - La': u'88.31', u'ICP - Li': u'25.39', u'ICP - Mg': u'5677.31', u'ICP - Na': u'2328.68', u'ICP - Ni': u'8.6', u'ICP - Sc': u'9.74', u'ICP - Sr': u'53.75', u'ICP - Ti': u'3030.14', u'ICP - V': u'39.88', u'ICP - Yb': u'3.83', u'ICP - Zn': u'88.53', u'Museum No.': u'706', u'NAA validation - As': u'6.9', u'NAA validation - Au': u'0', u'NAA validation - Ba': u'526', u'NAA validation - Br': u'0', u'NAA validation - Ca': u'0', u'NAA validation - Ce': u'128', u'NAA validation - Co': u'20.9', u'NAA validation - Cr': u'23.4', u'NAA validation - Cs': u'6.5', u'NAA validation - Eu': u'1.36', u'NAA validation - Fe': u'1.53', u'NAA validation - Hf': u'11', u'NAA validation - K': u'2.59', u'NAA validation - La': u'74.5', u'NAA validation - Lu': u'0.59', u'NAA validation - Na': u'0.244', u'NAA validation - Rb': u'154', u'NAA validation - Sb': u'0.49', u'NAA validation - Sc': u'11.4', u'NAA validation - Sm': u'7.8', u'NAA validation - Ta': u'2.74', u'NAA validation - Tb': u'1.2', u'NAA validation - Th': u'37.6', u'NAA validation - U': u'10.5', u'NAA validation - Yb': u'4', u'NAA validation - Zn': u'90.2', u'PIXE - Al(1014)': u'129585', u'PIXE - Ca': u'1601', u'PIXE - F(area)': u'183.9', u'PIXE - Fe': u'14970', u'PIXE - K': u'25205', u'PIXE - Li(478)': u'10.2', u'PIXE - Mg(585)': u'0.0001', u'PIXE - Mn': u'405', u'PIXE - Na(440)': u'2659.3', u'PIXE - Rb': u'153', u'PIXE - Si': u'434771', u'PIXE - Sr': u'44', u'PIXE - Ti': u'4227', u'PIXE - V': u'74', u'PIXE - Zr': u'312', u'Photograph No. - Located In Photographs Folder.': u'UNE 104', u'Rel: http://www.cidoc-crm.org/rdfs/cidoc-crm#P2.has_type': u'http://collection.britishmuseum.org/id/thesauri/x7402', u'Rel: http://www.cidoc-crm.org/rdfs/cidoc-crm#P45F.consists_of': u'http://collection.britishmuseum.org/id/thesauri/x10539', u'Sample Source Person': u'Eusebio Dizon', u'Sample Weight (g)': u'4.5', u'Vessel Part Sampled': u'base', u'Year': u'1600'}}
# list the URLs for the thumbnails
[result.get('thumbIcon') for result in results]
[u'http://artiraq.org/static/opencontext/stoneware-media/thumbs/photographs/Edited%20Copies/Copy%20(2)%20of%20une104%20copy.jpg', u'http://artiraq.org/static/opencontext/stoneware-media/thumbs/photographs/Edited%20Copies/UNE373%20copy.jpg', u'http://artiraq.org/static/opencontext/stoneware-media/thumbs/photographs/Edited%20Copies/une343%20copy.jpg', u'http://artiraq.org/static/opencontext/stoneware-media/thumbs/photographs/Edited%20Copies/une342%20copy.jpg', u'http://artiraq.org/static/opencontext/stoneware-media/thumbs/photographs/Edited%20Copies/une338%20copy.jpg', u'http://artiraq.org/static/opencontext/stoneware-media/thumbs/photographs/Edited%20Copies/une233%20copy.jpg', u'http://artiraq.org/static/opencontext/stoneware-media/thumbs/photographs/Edited%20Copies/une267%20copy.jpg', u'http://artiraq.org/static/opencontext/stoneware-media/thumbs/photographs/Edited%20Copies/une375%20copy.jpg', u'http://artiraq.org/static/opencontext/stoneware-media/thumbs/photographs/Edited%20Copies/UNE115%20copy.jpg', u'http://artiraq.org/static/opencontext/stoneware-media/thumbs/photographs/UNE149.JPG']
# do a quick display
from IPython.display import HTML
from jinja2 import Template
CSS = """
<style>
.wrap img {
margin-left: 0px;
margin-right: 0px;
display: inline-block;
}
</style>
"""
IMAGES_TEMPLATE = CSS + """
<div class="wrap">
{% for item in items %}<img title="{{item.label}}" src="{{item.thumbIcon}}"/>{% endfor %}
</div>
"""
template = Template(IMAGES_TEMPLATE)
HTML(template.render(items=results))
import requests
url = "http://opencontext.org/sets/.json"
r = requests.get(url)
r.json().keys()
[u'updated', u'sorting', u'numFound', u'facets', u'offset', u'geoCount', u'chronoTileFacets', u'summary', u'paging', u'qstring', u'published', u'results', u'paramCount', u'geoTileFacets']
r.json()['numFound']
840420
r.json()['paging']['prev']
False
# write a generator for all items in http://opencontext.org/sets/.json
import requests
def opencontext_items():
url = "http://opencontext.org/sets/.json"
more_items = True
while more_items:
r = requests.get(url)
for item in r.json()['results']:
yield item
url = r.json()['paging']['next']
if not url:
more_items = False
from itertools import islice
results = list(islice(opencontext_items(), 25))
HTML(template.render(items=results))
import requests
import lxml
from lxml import etree
url = "http://opencontext.org/projects/.atom"
r = requests.get(url)
doc = etree.fromstring(r.content)
doc
<Element {http://www.w3.org/2005/Atom}feed at 0x1050bad40>
# get list of titles
project_titles = [e.find('{http://www.w3.org/2005/Atom}title').text for e in doc.findall('{http://www.w3.org/2005/Atom}entry')]
for (i, title) in enumerate(project_titles):
print i+1, title
1 Faunal Data from Neolithic Menteşe: (Overview) 2 Kentucky Site Files: (Overview) 3 Illinois Site Files: (Overview) 4 Iowa Site Files: (Overview) 5 Indiana Site Files: (Overview) 6 Missouri Site Files: (Overview) 7 South Carolina SHPO: (Overview) 8 Georgia Archaeological Site File (GASF): (Overview) 9 Florida Site Files: (Overview) 10 Pyla-Koutsopetria Archaeological Project: (Overview) 11 Balance Pan Weights from Nippur: (Overview) 12 Osteometric Database of South American Camelids: (Overview) 13 Ceramics, Trade, Provenience and Geology: Cyprus in the Late Bronze Age: (Overview) 14 Archaeology of Mesoamerican Animals: (Overview) 15 Çatalhöyük Zooarchaeology: (Overview) 16 Çatalhöyük Area TP Zooarchaeology: (Overview) 17 Ilıpınar Zooarchaeology: (Overview) 18 Zooarchaeology of Neolithic Ulucak: (Overview) 19 Çukuriçi Höyük Zooarchaeology: (Overview) 20 Barçın Höyük Zooarchaeology: (Overview) 21 Köşk Höyük Faunal Data: (Overview) 22 Erbaba Höyük and Suberde Zooarchaeology: (Overview) 23 Mikt’sqaq Angayuk Finds: (Overview) 24 Asian Stoneware Jars: (Overview) 25 Zooarchaeology of Öküzini Cave: (Overview) 26 Zooarchaeology of Karain Cave B: (Overview) 27 West Stow West Zooarchaeology: (Overview) 28 Murlo: (Overview) 29 Hacksilber Project: (Overview) 30 Kenan Tepe: (Overview) 31 Rough Cilicia: (Overview) 32 Dhiban Excavation and Development Project: (Overview) 33 Tal-e Malyan Zooarchaeology: Tal-e Malyan Zooarchaeology 34 Zooarchaeology of Medieval Emden: (Overview) 35 Chogha Mish Fauna: (Overview) 36 Khirbat al-Mudayna al-Aliya: (Overview) 37 Dove Mountain Groundstone: (Overview) 38 Bade Museum: (Overview) 39 San Diego Archaeological Center: (Overview) 40 Presidio of San Francisco: (Overview) 41 Aegean Archaeomalacology: (Overview) 42 Petra Great Temple Excavations: (Overview) 43 Iraq Heritage Program: (Overview) 44 Lake Carlos Beach Site, 1992 and 1996: (Overview) 45 Corneal Ulceration in South East Asia: (Overview) 46 Harvard Peabody Mus. Zooarchaeology: (Overview) 47 Hazor: Zooarchaeology: (Overview) 48 Hayonim: Micromorphology: (Overview) 49 Geissenklosterle: Micromorphology: (Overview) 50 Pınarbaşı 1994: Animal Bones: (Overview) 51 Domuztepe Excavations: (Overview)