NEWT Python Tutorial

In the tutorial we go over some simple examples using python to communicate with the NEWT API to access NERSC. More details on NEWT can be found at http://newt.nersc.gov

We use the requests python module (See http://www.python-requests.org) for this tutorial since it really simplifies interactions over HTTP, and allows you to hold on to authenticated sessions.

In [1]:
from requests import Session

Let's initialize a session object

In [2]:
s = Session()

Now, we're going to do the simplest possible call to NEWT. This is the "Hello World Example"

In [3]:
r = s.get("https://newt.nersc.gov/newt")

Let's examine the content of the response

In [4]:
r.content
Out[4]:
'{"status": "OK", "output": {"text": "Welcome to NEWT", "version": "0.3.2"}, "error": ""}'

Now we check our current authentication status

In [5]:
r = s.get("https://newt.nersc.gov/newt/auth")
In [6]:
r.content
Out[6]:
'{"username": null, "session_lifetime": 0, "auth": false, "newt_sessionid": null}'

Now we authenticate. Just provide a dict with of the form {"username": "myusername, "password": "mypassword"}. Here mypassword will be your NERSC password and your MFA string combined. POST it to the https://newt.nersc.gov/newt/auth URL

In [7]:
password = 'XXXXXX' + 'YYYYYY' # XXXXXX is NERSC passwd, YYYYYY is NERSC MFA
r = s.post("https://newt.nersc.gov/newt/auth", {"username": "shreyas", "password": password})

Check our status code and the content to make sure it succeeded

In [8]:
r.status_code
Out[8]:
200
In [9]:
r.content
Out[9]:
'{"username": "shreyas", "session_lifetime": 43198, "auth": true, "newt_sessionid": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"}'

Now we create a new batch job on cori. Post the path for the SLURM job submission file on the target system. This should be a post request to https://newt.nersc.gov/newt/queue/cori/ using a dict of the form {"jobfile": "/path/to/file/on/cori"}. You can also post a "jobscript" (instead of a "jobfile") containing the raw contents of the script.

In [10]:
r = s.post("https://newt.nersc.gov/newt/queue/cori/", {"jobfile": "/project/projectdirs/osp/newt.sbatch"})

The JOB id is in the resulting "jobid" field. Note that the requests object has a json() method that will automatically convert json output into python objects

In [11]:
r.content
Out[11]:
'{"status": "OK", "error": "", "jobid": "1250312"}'
In [12]:
jobid = r.json()['jobid']

Let's use this jobid to query the queue

In [13]:
r = s.get("https://newt.nersc.gov/newt/queue/cori/%s" % jobid)
In [14]:
r.status_code
Out[14]:
200
In [15]:
r.content
Out[15]:
'{"status": "PD", "repo": "mpccc", "rank_bf": "N/A", "qos": "normal", "name": "newt.sbatch", "timeuse": "0:00", "hostname": "cori", "jobid": "1250312", "queue": "regular", "submittime": "2016-03-02T10:50:04", "reason": "Priority", "user": "shreyas", "memory": "0", "nodes": "1", "timereq": "1:00", "procs": "1", "rank_p": "5607"}'

Now let's get everything for user shreyas. Note you can query for any field in the job object with ?key=value

In [16]:
r = s.get("https://newt.nersc.gov/newt/queue/cori/?user=shreyas")
In [17]:
j = r.json()
In [18]:
len(j)
Out[18]:
2

Sample shreyas job (the first job in the returned list)

In [19]:
j[0]
Out[19]:
{u'hostname': u'cori',
 u'jobid': u'1250267',
 u'memory': u'0',
 u'name': u'newt.sbatch',
 u'nodes': u'1',
 u'procs': u'1',
 u'qos': u'normal',
 u'queue': u'regular',
 u'rank_bf': u'N/A',
 u'rank_p': u'5580',
 u'reason': u'Priority',
 u'repo': u'mpccc',
 u'status': u'PD',
 u'submittime': u'2016-03-02T10:38:01',
 u'timereq': u'1:00',
 u'timeuse': u'0:00',
 u'user': u'shreyas'}
In [ ]: