In the tutorial we go over some simple examples using python to communicate with the NEWT API to access NERSC. More details on NEWT can be found at http://newt.nersc.gov
We use the requests
python module (See http://www.python-requests.org) for this tutorial since it really simplifies interactions over HTTP, and allows you to hold on to authenticated sessions.
from requests import Session
Let's initialize a session object
s = Session()
Now, we're going to do the simplest possible call to NEWT. This is the "Hello World Example"
r = s.get("https://newt.nersc.gov/newt")
Let's examine the content of the response
r.content
Now we check our current authentication status
r = s.get("https://newt.nersc.gov/newt/auth")
r.content
Now we authenticate. Just provide a dict with of the form {"username": "myusername, "password": "mypassword"}
. Here mypassword will be your NERSC password and your MFA string combined. POST
it to the https://newt.nersc.gov/newt/auth URL
password = 'XXXXXX' + 'YYYYYY' # XXXXXX is NERSC passwd, YYYYYY is NERSC MFA
r = s.post("https://newt.nersc.gov/newt/auth", {"username": "shreyas", "password": password})
Check our status code and the content to make sure it succeeded
r.status_code
r.content
Now we create a new batch job on cori. Post the path for the SLURM job submission file on the target system. This should be a post
request to https://newt.nersc.gov/newt/queue/cori/ using a dict of the form {"jobfile": "/path/to/file/on/cori"}
. You can also post a "jobscript"
(instead of a "jobfile"
) containing the raw contents of the script.
r = s.post("https://newt.nersc.gov/newt/queue/cori/", {"jobfile": "/project/projectdirs/osp/newt.sbatch"})
The JOB id is in the resulting "jobid"
field. Note that the requests object has a json()
method that will automatically convert json output into python objects
r.content
jobid = r.json()['jobid']
Let's use this jobid to query the queue
r = s.get("https://newt.nersc.gov/newt/queue/cori/%s" % jobid)
r.status_code
r.content
Now let's get everything for user shreyas
. Note you can query for any field in the job object with ?key=value
r = s.get("https://newt.nersc.gov/newt/queue/cori/?user=shreyas")
j = r.json()
len(j)
Sample shreyas job (the first job in the returned list)
j[0]