1. notebooks
  2. ao-prov-modelling.ipynb

0. Initialisation

All components to run the code in this section:

In [1]:
%pylab inline
from datetime import datetime
import time
from prov.model import ProvDocument, Namespace
from provstore.api import Api

api = Api(username='', api_key='')
ao = Namespace('ao', 'https://provenance.ecs.soton.ac.uk/atomicorchid/ns#')
incident_types = {
    0: 'WaterSource',
    1: 'InfrastructureDamage',
    2: 'MedicalEmergency',
    3: 'Crime'
}
responder_types = {
    0: 'Medic',
    1: 'Firefighter',
    2: 'Soldier',
    3: 'Transporter'
}

Populating the interactive namespace from numpy and matplotlib
In [2]:
# Apply to all components
game_id = 1  # Set the game_id to aggreed value to be given by Nottingham

1. CrowdScanner Provenance

In [3]:
d = ProvDocument()
default_ns = 'https://provenance.ecs.soton.ac.uk/atomicorchid/data/%s/' % game_id
d.add_namespace(ao)
d.set_default_namespace(default_ns)
In [4]:
# submit the blank document to ProvStore. The actual data will go into bundles later
stored_document = api.document.create(d, name="Game%s CrowdScanner" % game_id)
# The following document_id needs to be stored and used each time the CrowdScanner submit provenance to ProvStore
document_id = stored_document.id
In [29]:
target_versions = {}  # Dictionary to record the current (latest) version of a target
posted_reports = {}  # Dictionary to record the reports that have been posted to ProvStore

1.1 First Update

In [30]:
timestamp = time.time()  # Record the timestamp at each update to generate unique identifiers

b = ProvDocument()  # Create a new document for this update
b.add_namespace(ao)
b.set_default_namespace(default_ns)
# cs to be used with all targets
cs = b.agent('agent/CrowdScanner', (('prov:type', 'ao:IBCCAlgo'), ('prov:type', 'prov:SoftwareAgent')))

startTime = datetime.fromtimestamp(timestamp)
endTime = startTime
activity = b.activity('activity/cs/update_report_%s' % timestamp, startTime, endTime)
activity.wasAssociatedWith(cs)
Out[30]:
<ProvActivity: activity/cs/update_report_1411042249.12>

Assuming the invalidadted targets are not included here, for each target:

In [31]:
tid, x, y, v = 35, 18.5576, -72.3, 0
target_attributes = {'ao:longitude':x, 'ao:latitude': y}
In [32]:
target_name = 'cs/target/%s' % tid
target_v = b.entity('cs/target/%s.%s' % (tid, v), target_attributes)
target_v.specializationOf(target_name)  # assert that this is a specialisation of the generic entity, using only name is OK
if tid not in target_versions:  # this is the fist time
    b.entity(target_name)  # create the generic entity for the target   
else:
    # no need to create the generic entity in subsequent updates
    target_v.wasDerivedFrom(target_versions[tid])
target_v.wasAttributedTo(cs)  # attributing the target to the agent (everytime)
target_v.wasGeneratedBy(activity)
target_versions[tid] = target_v   # update the current version of target #tid

Now recording the links from the targets to reports:

In [33]:
report_id, x, y, report_content, reporter_id = 98, 283.0, 220.0, 'La Plaza Hotel in Delmas 33', 2
In [34]:
if report_id not in posted_reports:  # this is the first time we've seen this report
    # Extracting report_id, x, y, report_content, reporter_id here
    reporter_name = 'agent/crowdreporter%s' % reporter_id
    b.agent(reporter_name, (('prov:type', 'ao:CrowdReporter'), ('prov:type', 'prov:Person')))
    report_attrs = {'ao:longitude':x, 'ao:latitude': y, 'ao:report': report_content}
    crowd_report = b.entity('cs/report%s' % report_id, report_attrs)
    crowd_report.wasAttributedTo(reporter_name)
    activity.used(crowd_report)
    posted_reports[report_id] = crowd_report

target_v.wasDerivedFrom(posted_reports[report_id])
Out[34]:
<ProvEntity: cs/target/35.0>
In [35]:
b.plot()

1.2 Each subsequent update

Similar to the above. However, the version of each target should be increased every time the target is updated to avoid self derivations like this: https://provenance.ecs.soton.ac.uk/atomicorchid/data/12/target/35.0/prov.svg (see target/35.1)

1.3 Invalidation of targets

For each target (identified by tid) in a list to invalidated (targets_to_invalidate)

In [ ]:
if tid in target_versions:  # we should have tid in target_versions, this is just to be sure
    target_v = target_versions[tid]
    b.wasInvalidatedBy(target_v, activity)

1.4 Submit provenance to ProvStore

To avoid creating multiple documents when the CrowdScanner is run, we created only a single document (at the very first step) to hold the provenance submitted by the application. Each provenance update will be put into a bundle. The bundle name needs to be unique inside a document, hence, we use the timestampe in the name as below.

In [36]:
# Create a bundle name to submit to the ProvStore document
bundle_identifier = 'bundle/csupdate/%s' % timestamp
bundle_identifier
Out[36]:
'bundle/csupdate/1411042249.12'
In [11]:
# Submitting the document created above to the store (as a bundle in the previouly created document)
api.add_bundle(document_id, b.serialize(), bundle_identifier)
Out[11]:
True

2. UAV Controller Provenance

In [50]:
d = ProvDocument()
default_ns = 'https://provenance.ecs.soton.ac.uk/atomicorchid/data/%s/' % game_id
d.add_namespace(ao)
d.set_default_namespace(default_ns)
In [51]:
# submit the blank document to ProvStore. The actual data will go into bundles later
stored_document = api.document.create(d, name="Game%s UAV Controller" % game_id)
# The following document_id needs to be stored and used each time the CrowdScanner submit provenance to ProvStore
document_id = stored_document.id

2.1 First time: Annotate the type of a target

In [58]:
# Getting the timestamp to generate unique identifiers
timestamp = time.time()

# Required information from Crowdscanner
cs_target_id = 4
cs_target_version = 2
cs_target_name = 'cs/target/%s.%s' % (cs_target_id, cs_target_version)

# User input
operator_name = 'uav_silver_commander'  # or 'uav_bronze_commander'
target_type_id = 3  # Crime
In [59]:
d = ProvDocument()
d.add_namespace(ao)
d.set_default_namespace(default_ns)
In [60]:
operator_id = 'agent/' + operator_name
d.agent(operator_id, (('prov:type', 'ao:UAVOperator'), ('prov:type', 'prov:Person')))

target_version = 0
target_name = 'uav/target/%s.%s' % (cs_target_id, target_version)
targe_type_attribute = ao[incident_types[target_type_id]]  # looking up the predefined incident's type name
d.entity(target_name, {'ao:asset_type': targe_type_attribute})

startTime = datetime.fromtimestamp(timestamp)
endTime = startTime
activity_id = 'activity/uav_verification/%s' % timestamp
d.activity(activity_id, startTime, endTime)
d.used(activity_id, cs_target_name)
d.wasAssociatedWith(activity_id, operator_id)
d.wasGeneratedBy(target_name, activity_id)

d.wasDerivedFrom(target_name, cs_target_name)
d.wasAttributedTo(target_name, operator_id)

d.plot()

2.2 Update the type of a target

We should run this only when the updated incident type is different that the previous type.

In [62]:
# Getting the timestamp to generate unique identifiers
timestamp = time.time()

# Required information
target_id = 4
target_version = 0
target_name_v0 = 'uav/target/%s.%s' % (target_id, target_version)

# User input
operator_name = 'uav_bronze_commander'  # or 'uav_silver_commander'
target_type_id = 0  # WaterSource
In [63]:
d = ProvDocument()
d.add_namespace(ao)
d.set_default_namespace(default_ns)
In [64]:
operator_id = 'agent/' + operator_name
d.agent(operator_id, (('prov:type', 'ao:UAVOperator'), ('prov:type', 'prov:Person')))

target_version += 1  # TODO: You need to update your internal version as well
target_name = 'uav/target/%s.%s' % (cs_target_id, target_version)
targe_type_attribute = ao[incident_types[target_type_id]]  # looking up the predefined incident's type name
d.entity(target_name, {'ao:asset_type': targe_type_attribute})

startTime = datetime.fromtimestamp(timestamp)
endTime = startTime
activity_id = 'activity/uav_verification/%s' % timestamp
d.activity(activity_id, startTime, endTime)
d.used(activity_id, target_name_v0)
d.wasAssociatedWith(activity_id, operator_id)
d.wasGeneratedBy(target_name, activity_id)
d.wasDerivedFrom(target_name, target_name_v0)
d.wasAttributedTo(target_name, operator_id)

d.wasInvalidatedBy(target_name_v0, activity_id)

d.plot()

2.3 UAV Operator manually adds a new target

In [67]:
# Getting the timestamp to generate unique identifiers
timestamp = time.time()

# Required information
target_id = 100

# User input
operator_name = 'uav_bronze_commander'  # or 'uav_silver_commander'
target_type_id = 1
In [68]:
d = ProvDocument()
d.add_namespace(ao)
d.set_default_namespace(default_ns)
In [69]:
operator_id = 'agent/' + operator_name
d.agent(operator_id, (('prov:type', 'ao:UAVOperator'), ('prov:type', 'prov:Person')))

target_version = 0  # TODO: Store this version number internally
target_name = 'uav/target/%s.%s' % (target_id, target_version)

targe_type_attribute = ao[incident_types[target_type_id]]  # looking up the predefined incident's type name
d.entity(target_name, {'ao:asset_type': targe_type_attribute})

startTime = datetime.fromtimestamp(timestamp)
endTime = startTime
activity_id = 'activity/uav_verification/%s' % timestamp
d.activity(activity_id, startTime, endTime)
d.wasAssociatedWith(activity_id, operator_id)

d.wasGeneratedBy(target_name, activity_id)
d.wasAttributedTo(target_name, operator_id)


d.plot()

2.4 Submit provenance to ProvStore

In [70]:
# submit d to ProvStore as a bundle 
# document_id = 28380  # This should have been saved previously
bundle_id = 'bundle/uav_update/%s' % timestamp
api.add_bundle(document_id, d.serialize(), bundle_id)
Out[70]:
True

3. Planner Provenance

In [5]:
b = d.bundle('planner')
planner = b.agent('planner_agent')
plan = b.entity('plan/k', {'prov:type': ao['Plan']})
p1 = d.entity('player1', {'prov:type': ao['Firefighter']})
p1_33 = b.entity('player1.33')
p1_33.specializationOf(p1)
p2 = b.entity('player2', {'prov:type': ao['Police']})
p2_35 = b.entity('player2.35')
p2_35.specializationOf(p2)
task = b.entity(
    'task/j',
    (('prov:type', ao['Task']), ('ao:target', target_v1), ('ao:assignee', p1), ('ao:assignee', p2))
)
plan.hadMember(task)

instruction_1 = b.entity('instruction/m.0', {'ao:recipient': p1, 'ao:target': target_v1})
instruction_2 = b.entity('instruction/n', {'ao:recipient': p2, 'ao:target': target_v1})
task.hadMember(instruction_1)
task.hadMember(instruction_2)

instruction_1.wasDerivedFrom(target_v1)
instruction_1.wasDerivedFrom(p1_33)
instruction_2.wasDerivedFrom(target_v1)
instruction_2.wasDerivedFrom(p2_35)

plan.wasAttributedTo(planner)
Out[5]:
<ProvEntity: plan/k>
In [6]:
b = d.bundle('headquarter')
hq = b.agent('hq')
confirmed_plan = b.entity('confirmed_plan/h', {'prov:type': ao['Plan']})
confirmed_plan.hadMember(task)
confirmed_plan.wasDerivedFrom(plan)
confirmed_plan.wasAttributedTo(hq)
Out[6]:
<ProvEntity: confirmed_plan/h>
In [7]:
b = d.bundle('player_1_confirmation')
ca = b.activity('confirmation/m')
p1_34 = b.entity('player1.34')
p1_34.wasDerivedFrom(p1_33)
p1_34.specializationOf(p1)
ca.wasAssociatedWith(p1_34)
confirmed_instruction_1 = b.entity('instruction/m.1', {'ao:status': ao['confirmed']})
confirmed_instruction_1.wasDerivedFrom(instruction_1, ca)
confirmed_instruction_1.wasAttributedTo(p1_34)
Out[7]:
<ProvEntity: instruction/m.1>
In [11]:
d.plot()

Couldn't import dot_parser, loading of dot files will not be possible.
In [9]:
d.flattened().plot()
In [2]:
import time
ts = time.time()
print ts

1410190345.56
In [ ]: