This notebook will download all 120k+ articles with abstracts from the PLOS API.
import pandas as pd
import numpy as np
import settings
import requests
import urllib
import time
from retrying import retry
import os
#adapted from Raymond's notebook
def plos_search(q,start=0,rows=100,fl=None, extras=None):
BASE_URL = 'http://api.plos.org/search'
DEFAULT_FL = ('abstract','author',
'id','journal','publication_date',
'score','title_display', 'subject','subject_level')
#removed elements: eissn, article_type
# fl indicates fields to return
# http://wiki.apache.org/solr/CommonQueryParameters#fl
if fl is None:
fl_ = ",".join(DEFAULT_FL)
else:
fl_ = ",".join(fl)
query = {'q':q,
'start':start,
'rows':rows,
'api_key':settings.PLOS_KEY,
'wt':'json',
'fl':fl_,
'fq': 'doc_type:full AND !article_type_facet:"Issue Image"'}
if extras is not None:
query.update(extras)
query_url = BASE_URL + "?" +urllib.urlencode(query)
r = requests.get(query_url)
return r
Need to make sure the calls do not exceed the following:
To be safe there will be a 15 second wait between each call:
r = plos_search(q='*')
tot_articles = r.json()['response']['numFound']
tot_articles
126718
With 118545 119637 126718 acticles total that means that we will have to perform ~1.2k API requests at 100 articles per request.
At 240 requests per hour it should take about 5.5 hours to get all the data needed.
This function will call the plos_search function every 15 seconds while incrementing the start number so that all of the articles can be pulled.
@retry(wait='exponential_sleep', wait_exponential_multiplier=1000, wait_exponential_max=10000, stop='stop_after_attempt', stop_max_attempt_number=200)
def data_request(end, start=0):
current_end = end - start
rows_per_query = 100
loops = (current_end/rows_per_query) + 1
for n in range(loops):
current_pickle = '../data/pickles/plos_pickle_{0}.pkl'.format(n)
#skips rows that have already been downloaded
if os.path.exists(current_pickle):
start += rows_per_query
continue
r = plos_search(q='subject:*', start=start, rows=rows_per_query)
#increment the start for the next request
start += rows_per_query
#store data before next call
data = r.json()['response']['docs']
new_df = pd.DataFrame(data)
#every request pickle the dataframe
new_df.to_pickle(current_pickle)
#prints status of job
print start, " articles pickled"
#wait a few seconds before the next loop
time.sleep(7)
print "DONE!"
Now we can run the function inputing the tot_articles as the end parameter.
data/pickles/
directory and it is empty.data/all_plos_df.pkl
is suitably backed up (or does not exist).First collection: crashed at 12100, 43800, 49500, 53500
Update 07-2014: What! No crashes!
all_plos_df = data_request(end=tot_articles)
100 articles pickled 200 articles pickled 300 articles pickled 400 articles pickled 500 articles pickled 600 articles pickled 700 articles pickled 800 articles pickled 900 articles pickled 1000 articles pickled 1100 articles pickled 1200 articles pickled 1300 articles pickled 1400 articles pickled 1500 articles pickled 1600 articles pickled 1700 articles pickled 1800 articles pickled 1900 articles pickled 2000 articles pickled 2100 articles pickled 2200 articles pickled 2300 articles pickled 2400 articles pickled 2500 articles pickled 2600 articles pickled 2700 articles pickled 2800 articles pickled 2900 articles pickled 3000 articles pickled 3100 articles pickled 3200 articles pickled 3300 articles pickled 3400 articles pickled 3500 articles pickled 3600 articles pickled 3700 articles pickled 3800 articles pickled 3900 articles pickled 4000 articles pickled 4100 articles pickled 4200 articles pickled 4300 articles pickled 4400 articles pickled 4500 articles pickled 4600 articles pickled 4700 articles pickled 4800 articles pickled 4900 articles pickled 5000 articles pickled 5100 articles pickled 5200 articles pickled 5300 articles pickled 5400 articles pickled 5500 articles pickled 5600 articles pickled 5700 articles pickled 5800 articles pickled 5900 articles pickled 6000 articles pickled 6100 articles pickled 6200 articles pickled 6300 articles pickled 6400 articles pickled 6500 articles pickled 6600 articles pickled 6700 articles pickled 6800 articles pickled 6900 articles pickled 7000 articles pickled 7100 articles pickled 7200 articles pickled 7300 articles pickled 7400 articles pickled 7500 articles pickled 7600 articles pickled 7700 articles pickled 7800 articles pickled 7900 articles pickled 8000 articles pickled 8100 articles pickled 8200 articles pickled 8300 articles pickled 8400 articles pickled 8500 articles pickled 8600 articles pickled 8700 articles pickled 8800 articles pickled 8900 articles pickled 9000 articles pickled 9100 articles pickled 9200 articles pickled 9300 articles pickled 9400 articles pickled 9500 articles pickled 9600 articles pickled 9700 articles pickled 9800 articles pickled 9900 articles pickled 10000 articles pickled 10100 articles pickled 10200 articles pickled 10300 articles pickled 10400 articles pickled 10500 articles pickled 10600 articles pickled 10700 articles pickled 10800 articles pickled 10900 articles pickled 11000 articles pickled 11100 articles pickled 11200 articles pickled 11300 articles pickled 11400 articles pickled 11500 articles pickled 11600 articles pickled 11700 articles pickled 11800 articles pickled 11900 articles pickled 12000 articles pickled 12100 articles pickled 12200 articles pickled 12300 articles pickled 12400 articles pickled 12500 articles pickled 12600 articles pickled 12700 articles pickled 12800 articles pickled 12900 articles pickled 13000 articles pickled 13100 articles pickled 13200 articles pickled 13300 articles pickled 13400 articles pickled 13500 articles pickled 13600 articles pickled 13700 articles pickled 13800 articles pickled 13900 articles pickled 14000 articles pickled 14100 articles pickled 14200 articles pickled 14300 articles pickled 14400 articles pickled 14500 articles pickled 14600 articles pickled 14700 articles pickled 14800 articles pickled 14900 articles pickled 15000 articles pickled 15100 articles pickled 15200 articles pickled 15300 articles pickled 15400 articles pickled 15500 articles pickled 15600 articles pickled 15700 articles pickled 15800 articles pickled 15900 articles pickled 16000 articles pickled 16100 articles pickled 16200 articles pickled 16300 articles pickled 16400 articles pickled 16500 articles pickled 16600 articles pickled 16700 articles pickled 16800 articles pickled 16900 articles pickled 17000 articles pickled 17100 articles pickled 17200 articles pickled 17300 articles pickled 17400 articles pickled 17500 articles pickled 17600 articles pickled 17700 articles pickled 17800 articles pickled 17900 articles pickled 18000 articles pickled 18100 articles pickled 18200 articles pickled 18300 articles pickled 18400 articles pickled 18500 articles pickled 18600 articles pickled 18700 articles pickled 18800 articles pickled 18900 articles pickled 19000 articles pickled 19100 articles pickled 19200 articles pickled 19300 articles pickled 19400 articles pickled 19500 articles pickled 19600 articles pickled 19700 articles pickled 19800 articles pickled 19900 articles pickled 20000 articles pickled 20100 articles pickled 20200 articles pickled 20300 articles pickled 20400 articles pickled 20500 articles pickled 20600 articles pickled 20700 articles pickled 20800 articles pickled 20900 articles pickled 21000 articles pickled 21100 articles pickled 21200 articles pickled 21300 articles pickled 21400 articles pickled 21500 articles pickled 21600 articles pickled 21700 articles pickled 21800 articles pickled 21900 articles pickled 22000 articles pickled 22100 articles pickled 22200 articles pickled 22300 articles pickled 22400 articles pickled 22500 articles pickled 22600 articles pickled 22700 articles pickled 22800 articles pickled 22900 articles pickled 23000 articles pickled 23100 articles pickled 23200 articles pickled 23300 articles pickled 23400 articles pickled 23500 articles pickled 23600 articles pickled 23700 articles pickled 23800 articles pickled 23900 articles pickled 24000 articles pickled 24100 articles pickled 24200 articles pickled 24300 articles pickled 24400 articles pickled 24500 articles pickled 24600 articles pickled 24700 articles pickled 24800 articles pickled 24900 articles pickled 25000 articles pickled 25100 articles pickled 25200 articles pickled 25300 articles pickled 25400 articles pickled 25500 articles pickled 25600 articles pickled 25700 articles pickled 25800 articles pickled 25900 articles pickled 26000 articles pickled 26100 articles pickled 26200 articles pickled 26300 articles pickled 26400 articles pickled 26500 articles pickled 26600 articles pickled 26700 articles pickled 26800 articles pickled 26900 articles pickled 27000 articles pickled 27100 articles pickled 27200 articles pickled 27300 articles pickled 27400 articles pickled 27500 articles pickled 27600 articles pickled 27700 articles pickled 27800 articles pickled 27900 articles pickled 28000 articles pickled 28100 articles pickled 28200 articles pickled 28300 articles pickled 28400 articles pickled 28500 articles pickled 28600 articles pickled 28700 articles pickled 28800 articles pickled 28900 articles pickled 29000 articles pickled 29100 articles pickled 29200 articles pickled 29300 articles pickled 29400 articles pickled 29500 articles pickled 29600 articles pickled 29700 articles pickled 29800 articles pickled 29900 articles pickled 30000 articles pickled 30100 articles pickled 30200 articles pickled 30300 articles pickled 30400 articles pickled 30500 articles pickled 30600 articles pickled 30700 articles pickled 30800 articles pickled 30900 articles pickled 31000 articles pickled 31100 articles pickled 31200 articles pickled 31300 articles pickled 31400 articles pickled 31500 articles pickled 31600 articles pickled 31700 articles pickled 31800 articles pickled 31900 articles pickled 32000 articles pickled 32100 articles pickled 32200 articles pickled 32300 articles pickled 32400 articles pickled 32500 articles pickled 32600 articles pickled 32700 articles pickled 32800 articles pickled 32900 articles pickled 33000 articles pickled 33100 articles pickled 33200 articles pickled 33300 articles pickled 33400 articles pickled 33500 articles pickled 33600 articles pickled 33700 articles pickled 33800 articles pickled 33900 articles pickled 34000 articles pickled 34100 articles pickled 34200 articles pickled 34300 articles pickled 34400 articles pickled 34500 articles pickled 34600 articles pickled 34700 articles pickled 34800 articles pickled 34900 articles pickled 35000 articles pickled 35100 articles pickled 35200 articles pickled 35300 articles pickled 35400 articles pickled 35500 articles pickled 35600 articles pickled 35700 articles pickled 35800 articles pickled 35900 articles pickled 36000 articles pickled 36100 articles pickled 36200 articles pickled 36300 articles pickled 36400 articles pickled 36500 articles pickled 36600 articles pickled 36700 articles pickled 36800 articles pickled 36900 articles pickled 37000 articles pickled 37100 articles pickled 37200 articles pickled 37300 articles pickled 37400 articles pickled 37500 articles pickled 37600 articles pickled 37700 articles pickled 37800 articles pickled 37900 articles pickled 38000 articles pickled 38100 articles pickled 38200 articles pickled 38300 articles pickled 38400 articles pickled 38500 articles pickled 38600 articles pickled 38700 articles pickled 38800 articles pickled 38900 articles pickled 39000 articles pickled 39100 articles pickled 39200 articles pickled 39300 articles pickled 39400 articles pickled 39500 articles pickled 39600 articles pickled 39700 articles pickled 39800 articles pickled 39900 articles pickled 40000 articles pickled 40100 articles pickled 40200 articles pickled 40300 articles pickled 40400 articles pickled 40500 articles pickled 40600 articles pickled 40700 articles pickled 40800 articles pickled 40900 articles pickled 41000 articles pickled 41100 articles pickled 41200 articles pickled 41300 articles pickled 41400 articles pickled 41500 articles pickled 41600 articles pickled 41700 articles pickled 41800 articles pickled 41900 articles pickled 42000 articles pickled 42100 articles pickled 42200 articles pickled 42300 articles pickled 42400 articles pickled 42500 articles pickled 42600 articles pickled 42700 articles pickled 42800 articles pickled 42900 articles pickled 43000 articles pickled 43100 articles pickled 43200 articles pickled 43300 articles pickled 43400 articles pickled 43500 articles pickled 43600 articles pickled 43700 articles pickled 43800 articles pickled 43900 articles pickled 44000 articles pickled 44100 articles pickled 44200 articles pickled 44300 articles pickled 44400 articles pickled 44500 articles pickled 44600 articles pickled 44700 articles pickled 44800 articles pickled 44900 articles pickled 45000 articles pickled 45100 articles pickled 45200 articles pickled 45300 articles pickled 45400 articles pickled 45500 articles pickled 45600 articles pickled 45700 articles pickled 45800 articles pickled 45900 articles pickled 46000 articles pickled 46100 articles pickled 46200 articles pickled 46300 articles pickled 46400 articles pickled 46500 articles pickled 46600 articles pickled 46700 articles pickled 46800 articles pickled 46900 articles pickled 47000 articles pickled 47100 articles pickled 47200 articles pickled 47300 articles pickled 47400 articles pickled 47500 articles pickled 47600 articles pickled 47700 articles pickled 47800 articles pickled 47900 articles pickled 48000 articles pickled 48100 articles pickled 48200 articles pickled 48300 articles pickled 48400 articles pickled 48500 articles pickled 48600 articles pickled 48700 articles pickled 48800 articles pickled 48900 articles pickled 49000 articles pickled 49100 articles pickled 49200 articles pickled 49300 articles pickled 49400 articles pickled 49500 articles pickled 49600 articles pickled 49700 articles pickled 49800 articles pickled 49900 articles pickled 50000 articles pickled 50100 articles pickled 50200 articles pickled 50300 articles pickled 50400 articles pickled 50500 articles pickled 50600 articles pickled 50700 articles pickled 50800 articles pickled 50900 articles pickled 51000 articles pickled 51100 articles pickled 51200 articles pickled 51300 articles pickled 51400 articles pickled 51500 articles pickled 51600 articles pickled 51700 articles pickled 51800 articles pickled 51900 articles pickled 52000 articles pickled 52100 articles pickled 52200 articles pickled 52300 articles pickled 52400 articles pickled 52500 articles pickled 52600 articles pickled 52700 articles pickled 52800 articles pickled 52900 articles pickled 53000 articles pickled 53100 articles pickled 53200 articles pickled 53300 articles pickled 53400 articles pickled 53500 articles pickled 53600 articles pickled 53700 articles pickled 53800 articles pickled 53900 articles pickled 54000 articles pickled 54100 articles pickled 54200 articles pickled 54300 articles pickled 54400 articles pickled 54500 articles pickled 54600 articles pickled 54700 articles pickled 54800 articles pickled 54900 articles pickled 55000 articles pickled 55100 articles pickled 55200 articles pickled 55300 articles pickled 55400 articles pickled 55500 articles pickled 55600 articles pickled 55700 articles pickled 55800 articles pickled 55900 articles pickled 56000 articles pickled 56100 articles pickled 56200 articles pickled 56300 articles pickled 56400 articles pickled 56500 articles pickled 56600 articles pickled 56700 articles pickled 56800 articles pickled 56900 articles pickled 57000 articles pickled 57100 articles pickled 57200 articles pickled 57300 articles pickled 57400 articles pickled 57500 articles pickled 57600 articles pickled 57700 articles pickled 57800 articles pickled 57900 articles pickled 58000 articles pickled 58100 articles pickled 58200 articles pickled 58300 articles pickled 58400 articles pickled 58500 articles pickled 58600 articles pickled 58700 articles pickled 58800 articles pickled 58900 articles pickled 59000 articles pickled 59100 articles pickled 59200 articles pickled 59300 articles pickled 59400 articles pickled 59500 articles pickled 59600 articles pickled 59700 articles pickled 59800 articles pickled 59900 articles pickled 60000 articles pickled 60100 articles pickled 60200 articles pickled 60300 articles pickled 60400 articles pickled 60500 articles pickled 60600 articles pickled 60700 articles pickled 60800 articles pickled 60900 articles pickled 61000 articles pickled 61100 articles pickled 61200 articles pickled 61300 articles pickled 61400 articles pickled 61500 articles pickled 61600 articles pickled 61700 articles pickled 61800 articles pickled 61900 articles pickled 62000 articles pickled 62100 articles pickled 62200 articles pickled 62300 articles pickled 62400 articles pickled 62500 articles pickled 62600 articles pickled 62700 articles pickled 62800 articles pickled 62900 articles pickled 63000 articles pickled 63100 articles pickled 63200 articles pickled 63300 articles pickled 63400 articles pickled 63500 articles pickled 63600 articles pickled 63700 articles pickled 63800 articles pickled 63900 articles pickled 64000 articles pickled 64100 articles pickled 64200 articles pickled 64300 articles pickled 64400 articles pickled 64500 articles pickled 64600 articles pickled 64700 articles pickled 64800 articles pickled 64900 articles pickled 65000 articles pickled 65100 articles pickled 65200 articles pickled 65300 articles pickled 65400 articles pickled 65500 articles pickled 65600 articles pickled 65700 articles pickled 65800 articles pickled 65900 articles pickled 66000 articles pickled 66100 articles pickled 66200 articles pickled 66300 articles pickled 66400 articles pickled 66500 articles pickled 66600 articles pickled 66700 articles pickled 66800 articles pickled 66900 articles pickled 67000 articles pickled 67100 articles pickled 67200 articles pickled 67300 articles pickled 67400 articles pickled 67500 articles pickled 67600 articles pickled 67700 articles pickled 67800 articles pickled 67900 articles pickled 68000 articles pickled 68100 articles pickled 68200 articles pickled 68300 articles pickled 68400 articles pickled 68500 articles pickled 68600 articles pickled 68700 articles pickled 68800 articles pickled 68900 articles pickled 69000 articles pickled 69100 articles pickled 69200 articles pickled 69300 articles pickled 69400 articles pickled 69500 articles pickled 69600 articles pickled 69700 articles pickled 69800 articles pickled 69900 articles pickled 70000 articles pickled 70100 articles pickled 70200 articles pickled 70300 articles pickled 70400 articles pickled 70500 articles pickled 70600 articles pickled 70700 articles pickled 70800 articles pickled 70900 articles pickled 71000 articles pickled 71100 articles pickled 71200 articles pickled 71300 articles pickled 71400 articles pickled 71500 articles pickled 71600 articles pickled 71700 articles pickled 71800 articles pickled 71900 articles pickled 72000 articles pickled 72100 articles pickled 72200 articles pickled 72300 articles pickled 72400 articles pickled 72500 articles pickled 72600 articles pickled 72700 articles pickled 72800 articles pickled 72900 articles pickled 73000 articles pickled 73100 articles pickled 73200 articles pickled 73300 articles pickled 73400 articles pickled 73500 articles pickled 73600 articles pickled 73700 articles pickled 73800 articles pickled 73900 articles pickled 74000 articles pickled 74100 articles pickled 74200 articles pickled 74300 articles pickled 74400 articles pickled 74500 articles pickled 74600 articles pickled 74700 articles pickled 74800 articles pickled 74900 articles pickled 75000 articles pickled 75100 articles pickled 75200 articles pickled 75300 articles pickled 75400 articles pickled 75500 articles pickled 75600 articles pickled 75700 articles pickled 75800 articles pickled 75900 articles pickled 76000 articles pickled 76100 articles pickled 76200 articles pickled 76300 articles pickled 76400 articles pickled 76500 articles pickled 76600 articles pickled 76700 articles pickled 76800 articles pickled 76900 articles pickled 77000 articles pickled 77100 articles pickled 77200 articles pickled 77300 articles pickled 77400 articles pickled 77500 articles pickled 77600 articles pickled 77700 articles pickled 77800 articles pickled 77900 articles pickled 78000 articles pickled 78100 articles pickled 78200 articles pickled 78300 articles pickled 78400 articles pickled 78500 articles pickled 78600 articles pickled 78700 articles pickled 78800 articles pickled 78900 articles pickled 79000 articles pickled 79100 articles pickled 79200 articles pickled 79300 articles pickled 79400 articles pickled 79500 articles pickled 79600 articles pickled 79700 articles pickled 79800 articles pickled 79900 articles pickled 80000 articles pickled 80100 articles pickled 80200 articles pickled 80300 articles pickled 80400 articles pickled 80500 articles pickled 80600 articles pickled 80700 articles pickled 80800 articles pickled 80900 articles pickled 81000 articles pickled 81100 articles pickled 81200 articles pickled 81300 articles pickled 81400 articles pickled 81500 articles pickled 81600 articles pickled 81700 articles pickled 81800 articles pickled 81900 articles pickled 82000 articles pickled 82100 articles pickled 82200 articles pickled 82300 articles pickled 82400 articles pickled 82500 articles pickled 82600 articles pickled 82700 articles pickled 82800 articles pickled 82900 articles pickled 83000 articles pickled 83100 articles pickled 83200 articles pickled 83300 articles pickled 83400 articles pickled 83500 articles pickled 83600 articles pickled 83700 articles pickled 83800 articles pickled 83900 articles pickled 84000 articles pickled 84100 articles pickled 84200 articles pickled 84300 articles pickled 84400 articles pickled 84500 articles pickled 84600 articles pickled 84700 articles pickled 84800 articles pickled 84900 articles pickled 85000 articles pickled 85100 articles pickled 85200 articles pickled 85300 articles pickled 85400 articles pickled 85500 articles pickled 85600 articles pickled 85700 articles pickled 85800 articles pickled 85900 articles pickled 86000 articles pickled 86100 articles pickled 86200 articles pickled 86300 articles pickled 86400 articles pickled 86500 articles pickled 86600 articles pickled 86700 articles pickled 86800 articles pickled 86900 articles pickled 87000 articles pickled 87100 articles pickled 87200 articles pickled 87300 articles pickled 87400 articles pickled 87500 articles pickled 87600 articles pickled 87700 articles pickled 87800 articles pickled 87900 articles pickled 88000 articles pickled 88100 articles pickled 88200 articles pickled 88300 articles pickled 88400 articles pickled 88500 articles pickled 88600 articles pickled 88700 articles pickled 88800 articles pickled 88900 articles pickled 89000 articles pickled 89100 articles pickled 89200 articles pickled 89300 articles pickled 89400 articles pickled 89500 articles pickled 89600 articles pickled 89700 articles pickled 89800 articles pickled 89900 articles pickled 90000 articles pickled 90100 articles pickled 90200 articles pickled 90300 articles pickled 90400 articles pickled 90500 articles pickled 90600 articles pickled 90700 articles pickled 90800 articles pickled 90900 articles pickled 91000 articles pickled 91100 articles pickled 91200 articles pickled 91300 articles pickled 91400 articles pickled 91500 articles pickled 91600 articles pickled 91700 articles pickled 91800 articles pickled 91900 articles pickled 92000 articles pickled 92100 articles pickled 92200 articles pickled 92300 articles pickled 92400 articles pickled 92500 articles pickled 92600 articles pickled 92700 articles pickled 92800 articles pickled 92900 articles pickled 93000 articles pickled 93100 articles pickled 93200 articles pickled 93300 articles pickled 93400 articles pickled 93500 articles pickled 93600 articles pickled 93700 articles pickled 93800 articles pickled 93900 articles pickled 94000 articles pickled 94100 articles pickled 94200 articles pickled 94300 articles pickled 94400 articles pickled 94500 articles pickled 94600 articles pickled 94700 articles pickled 94800 articles pickled 94900 articles pickled 95000 articles pickled 95100 articles pickled 95200 articles pickled 95300 articles pickled 95400 articles pickled 95500 articles pickled 95600 articles pickled 95700 articles pickled 95800 articles pickled 95900 articles pickled 96000 articles pickled 96100 articles pickled 96200 articles pickled 96300 articles pickled 96400 articles pickled 96500 articles pickled 96600 articles pickled 96700 articles pickled 96800 articles pickled 96900 articles pickled 97000 articles pickled 97100 articles pickled 97200 articles pickled 97300 articles pickled 97400 articles pickled 97500 articles pickled 97600 articles pickled 97700 articles pickled 97800 articles pickled 97900 articles pickled 98000 articles pickled 98100 articles pickled 98200 articles pickled 98300 articles pickled 98400 articles pickled 98500 articles pickled 98600 articles pickled 98700 articles pickled 98800 articles pickled 98900 articles pickled 99000 articles pickled 99100 articles pickled 99200 articles pickled 99300 articles pickled 99400 articles pickled 99500 articles pickled 99600 articles pickled 99700 articles pickled 99800 articles pickled 99900 articles pickled 100000 articles pickled 100100 articles pickled 100200 articles pickled 100300 articles pickled 100400 articles pickled 100500 articles pickled 100600 articles pickled 100700 articles pickled 100800 articles pickled 100900 articles pickled 101000 articles pickled 101100 articles pickled 101200 articles pickled 101300 articles pickled 101400 articles pickled 101500 articles pickled 101600 articles pickled 101700 articles pickled 101800 articles pickled 101900 articles pickled 102000 articles pickled 102100 articles pickled 102200 articles pickled 102300 articles pickled 102400 articles pickled 102500 articles pickled 102600 articles pickled 102700 articles pickled 102800 articles pickled 102900 articles pickled 103000 articles pickled 103100 articles pickled 103200 articles pickled 103300 articles pickled 103400 articles pickled 103500 articles pickled 103600 articles pickled 103700 articles pickled 103800 articles pickled 103900 articles pickled 104000 articles pickled 104100 articles pickled 104200 articles pickled 104300 articles pickled 104400 articles pickled 104500 articles pickled 104600 articles pickled 104700 articles pickled 104800 articles pickled 104900 articles pickled 105000 articles pickled 105100 articles pickled 105200 articles pickled 105300 articles pickled 105400 articles pickled 105500 articles pickled 105600 articles pickled 105700 articles pickled 105800 articles pickled 105900 articles pickled 106000 articles pickled 106100 articles pickled 106200 articles pickled 106300 articles pickled 106400 articles pickled 106500 articles pickled 106600 articles pickled 106700 articles pickled 106800 articles pickled 106900 articles pickled 107000 articles pickled 107100 articles pickled 107200 articles pickled 107300 articles pickled 107400 articles pickled 107500 articles pickled 107600 articles pickled 107700 articles pickled 107800 articles pickled 107900 articles pickled 108000 articles pickled 108100 articles pickled 108200 articles pickled 108300 articles pickled 108400 articles pickled 108500 articles pickled 108600 articles pickled 108700 articles pickled 108800 articles pickled 108900 articles pickled 109000 articles pickled 109100 articles pickled 109200 articles pickled 109300 articles pickled 109400 articles pickled 109500 articles pickled 109600 articles pickled 109700 articles pickled 109800 articles pickled 109900 articles pickled 110000 articles pickled 110100 articles pickled 110200 articles pickled 110300 articles pickled 110400 articles pickled 110500 articles pickled 110600 articles pickled 110700 articles pickled 110800 articles pickled 110900 articles pickled 111000 articles pickled 111100 articles pickled 111200 articles pickled 111300 articles pickled 111400 articles pickled 111500 articles pickled 111600 articles pickled 111700 articles pickled 111800 articles pickled 111900 articles pickled 112000 articles pickled 112100 articles pickled 112200 articles pickled 112300 articles pickled 112400 articles pickled 112500 articles pickled 112600 articles pickled 112700 articles pickled 112800 articles pickled 112900 articles pickled 113000 articles pickled 113100 articles pickled 113200 articles pickled 113300 articles pickled 113400 articles pickled 113500 articles pickled 113600 articles pickled 113700 articles pickled 113800 articles pickled 113900 articles pickled 114000 articles pickled 114100 articles pickled 114200 articles pickled 114300 articles pickled 114400 articles pickled 114500 articles pickled 114600 articles pickled 114700 articles pickled 114800 articles pickled 114900 articles pickled 115000 articles pickled 115100 articles pickled 115200 articles pickled 115300 articles pickled 115400 articles pickled 115500 articles pickled 115600 articles pickled 115700 articles pickled 115800 articles pickled 115900 articles pickled 116000 articles pickled 116100 articles pickled 116200 articles pickled 116300 articles pickled 116400 articles pickled 116500 articles pickled 116600 articles pickled 116700 articles pickled 116800 articles pickled 116900 articles pickled 117000 articles pickled 117100 articles pickled 117200 articles pickled 117300 articles pickled 117400 articles pickled 117500 articles pickled 117600 articles pickled 117700 articles pickled 117800 articles pickled 117900 articles pickled 118000 articles pickled 118100 articles pickled 118200 articles pickled 118300 articles pickled 118400 articles pickled 118500 articles pickled 118600 articles pickled 118700 articles pickled 118800 articles pickled 118900 articles pickled 119000 articles pickled 119100 articles pickled 119200 articles pickled 119300 articles pickled 119400 articles pickled 119500 articles pickled 119600 articles pickled 119700 articles pickled 119800 articles pickled 119900 articles pickled 120000 articles pickled 120100 articles pickled 120200 articles pickled 120300 articles pickled 120400 articles pickled 120500 articles pickled 120600 articles pickled 120700 articles pickled 120800 articles pickled 120900 articles pickled 121000 articles pickled 121100 articles pickled 121200 articles pickled 121300 articles pickled 121400 articles pickled 121500 articles pickled 121600 articles pickled 121700 articles pickled 121800 articles pickled 121900 articles pickled 122000 articles pickled 122100 articles pickled 122200 articles pickled 122300 articles pickled 122400 articles pickled 122500 articles pickled 122600 articles pickled 122700 articles pickled 122800 articles pickled 122900 articles pickled 123000 articles pickled 123100 articles pickled 123200 articles pickled 123300 articles pickled 123400 articles pickled 123500 articles pickled 123600 articles pickled 123700 articles pickled 123800 articles pickled 123900 articles pickled 124000 articles pickled 124100 articles pickled 124200 articles pickled 124300 articles pickled 124400 articles pickled 124500 articles pickled 124600 articles pickled 124700 articles pickled 124800 articles pickled 124900 articles pickled 125000 articles pickled 125100 articles pickled 125200 articles pickled 125300 articles pickled 125400 articles pickled 125500 articles pickled 125600 articles pickled 125700 articles pickled 125800 articles pickled 125900 articles pickled 126000 articles pickled 126100 articles pickled 126200 articles pickled 126300 articles pickled 126400 articles pickled 126500 articles pickled 126600 articles pickled 126700 articles pickled 126800 articles pickled DONE!
#concatenates pickled files
pieces = []
for i in range(0,1267): # replace end of range(0,x) with the number of the last file!
path = '../data/pickles/plos_pickle_{0}.pkl'.format(i)
frame = pd.read_pickle(path)
pieces.append(frame)
all_plos_df_for_export = pd.concat(pieces)
all_plos_df_for_export.to_pickle('../data/all_plos_df.pkl')
all_plos_df = pd.read_pickle('../data/all_plos_df.pkl')
# total number of articles (originally 115489)
len(all_plos_df)
121450
all_plos_df.sort(columns = 'id').head()
abstract | author | id | journal | publication_date | score | subject | title_display | |
---|---|---|---|---|---|---|---|---|
27 | [\n During lymphocyte development, V(D)... | [Alfred Ian Lee, Sebastian D Fugmann, Lindsay ... | 10.1371/journal.pbio.0000001 | PLoS Biology | 2003-10-13T00:00:00Z | 1 | [/Biology and life sciences/Molecular biology/... | A Functional Analysis of the Spacer of V(D)J R... |
59 | [\n Because of the constant threat pose... | [David Wang, Anatoly Urisman, Yu-Tsueng Liu, M... | 10.1371/journal.pbio.0000002 | PLoS Biology | 2003-11-17T00:00:00Z | 1 | [/Biology and life sciences/Molecular biology/... | Viral Discovery and Sequence Recovery Using DN... |
65 | [] | NaN | 10.1371/journal.pbio.0000003 | PLoS Biology | 2003-11-17T00:00:00Z | 1 | [/Biology and life sciences/Organisms/Viruses/... | Gene Chip for Viral Discovery |
51 | [] | NaN | 10.1371/journal.pbio.0000004 | PLoS Biology | 2003-10-13T00:00:00Z | 1 | [/Medicine and health sciences/Immunology/Immu... | Functional Analysis of RSS Spacers |
69 | [\n Plasmodium falciparum is the causat... | [Zbynek Bozdech, Manuel Llinás, Brian Lee Pull... | 10.1371/journal.pbio.0000005 | PLoS Biology | 2003-08-18T00:00:00Z | 1 | [/Biology and life sciences/Biochemistry/Prote... | The Transcriptome of the Intraerythrocytic Dev... |
5 rows × 8 columns
len(list(all_plos_df.id))
121450
print list(all_plos_df.subject)[0]
[u'/Medicine and health sciences/Pathology and laboratory medicine/Signs and symptoms/Fatigue', u'/Biology and life sciences/Anatomy/Musculoskeletal system/Muscles/Muscle fibers', u'/Medicine and health sciences/Physiology/Muscle physiology/Muscle contraction', u'/Biology and life sciences/Cell biology/Cellular types/Animal cells/Muscle fibers', u'/Biology and life sciences/Physiology/Muscle physiology/Muscle functions', u'/Medicine and health sciences/Anatomy/Musculoskeletal system/Muscles/Skeletal muscles', u'/Biology and life sciences/Anatomy/Musculoskeletal system/Muscles/Skeletal muscles', u'/Medicine and health sciences/Physiology/Muscle physiology/Muscle functions', u'/Biology and life sciences/Cell biology/Cellular types/Animal cells/Muscle fibers/Skeletal muscle fibers', u'/Biology and life sciences/Anatomy/Musculoskeletal system/Muscles/Muscle fibers/Skeletal muscle fibers', u'/Biology and life sciences/Physiology/Physiological processes/Homeostasis', u'/Biology and life sciences/Physiology/Muscle physiology/Muscle contraction', u'/Medicine and health sciences/Physiology/Physiological processes/Homeostasis', u'/Research and analysis methods/Imaging techniques/Fluorescence imaging', u'/Medicine and health sciences/Anatomy/Musculoskeletal system/Muscles/Muscle fibers', u'/Medicine and health sciences/Anatomy/Musculoskeletal system/Muscles/Muscle fibers/Skeletal muscle fibers']
all_plos_df.tail()
abstract | author | id | journal | publication_date | score | subject | title_display | |
---|---|---|---|---|---|---|---|---|
45 | [Background: Besides having an impact on human... | [Anna Puig-Oliveras, Yuliaxis Ramayo-Caldas, J... | 10.1371/journal.pone.0099720 | PLoS ONE | 2014-06-13T00:00:00Z | 1 | [/Biology and life sciences/Biochemistry/Lipid... | Differences in Muscle Transcriptome among Pigs... |
46 | [\nThe nuclear receptors and xenosensors const... | [Marianne Mathäs, Christian Nußhag , Oliver Bu... | 10.1371/journal.pone.0096263 | PLoS ONE | 2014-05-05T00:00:00Z | 1 | [/Biology and life sciences/Biochemistry/Prote... | Structural and Functional Similarity of Amphib... |
47 | [Background: Body dissatisfaction is the most ... | [Blanca Ortega-Roldán, Sonia Rodríguez-Ruiz, P... | 10.1371/journal.pone.0102595 | PLoS ONE | 2014-07-18T00:00:00Z | 1 | [/Medicine and health sciences/Anatomy/Integum... | The Emotional and Attentional Impact of Exposu... |
48 | [Objective: Maternal mortality (MM) is a core ... | [Juliana C Giordano, Mary A Parpinelli, Jose G... | 10.1371/journal.pone.0097401 | PLoS ONE | 2014-05-13T00:00:00Z | 1 | [/Medicine and health sciences/Women's health/... | The Burden of Eclampsia: Results from a Multic... |
49 | [\nOur previous studies showed positive correl... | [Mohammed M H Al-Gayyar, Barbara A Mysona, Sur... | 10.1371/journal.pone.0054692 | PLoS ONE | 2013-01-24T00:00:00Z | 1 | [/Biology and life sciences/Biochemistry/Enzym... | Diabetes and Overexpression of proNGF Cause Re... |
5 rows × 8 columns
This is making sure that the retry decorator works
import random
@retry
def do_something_unreliable():
if random.randint(0, 2) > 1:
raise IOError("Broken sauce, everything is hosed!!!111one")
else:
return "Awesome sauce!"
print do_something_unreliable()
Awesome sauce!