With increasing amounts of data being published on the web via APIs, one thing we may be tempted to do is build our own databases around it, subject to data licensing restrictions, of course...
In order to build our own database, we will need to make repeated calls to the API. This example shows how to create a simple MongDB database containing programme details for a particular BBC series.
The process is as follows:
#The BBC Programmes data is currently available as a JSON feed
import requests
import json
#We can get a list of the episodes associated with a series from the the episodes/player page for the series
#seriesStub='http://www.bbc.co.uk/programmes/b006qshd/episodes/player.json?page=' #More or Less, Radio 4
seriesStub='http://www.bbc.co.uk/programmes/p00msxfl/episodes/player.json?page=' #More or Less, World Service
#The following lambda function gives the URL for a particular episode based on its programme ID
programmeURL= lambda uid: "http://www.bbc.co.uk/programmes/"+uid+".json"
def getData(url):
''' A simple function to request the JSON data from a URL and return it as a dict '''
resp = requests.get(url)
data=json.loads(resp.content.decode('utf8'))
return data
#Test the function
data=getData(seriesStub)
data
{'total': 134, 'episodes': [{'programme': {'programme': {'pid': 'p00msxfl', 'title': 'More or Less', 'position': None, 'type': 'brand', 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01cnq9j'}, 'first_broadcast_date': '2012-01-13T23:50:00Z', 'expected_child_count': None}, 'pid': 'p023276m', 'media': {'format': 'audio', 'availability': 'Available to listen', 'expires': None}, 'position': None, 'has_clips': False, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p02307gn'}, 'is_available_mediaset_pc_sd': True, 'has_segment_events': False, 'display_titles': {'title': 'The Prevalence of Paedophilia', 'subtitle': ''}, 'has_medium_or_long_synopsis': True, 'has_related_links': False, 'media_type': 'audio', 'title': 'The Prevalence of Paedophilia', 'is_legacy_media': False, 'type': 'episode', 'short_synopsis': 'If 2% of Catholic clergy are paedophiles how does this compare to the rest of society?', 'actual_start': '2014-07-25T20:05:10+01:00', 'first_broadcast_date': '2014-07-25T19:50:00+01:00', 'duration': 660}}, {'programme': {'programme': {'pid': 'p00msxfl', 'title': 'More or Less', 'position': None, 'type': 'brand', 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01cnq9j'}, 'first_broadcast_date': '2012-01-13T23:50:00Z', 'expected_child_count': None}, 'pid': 'p022m7bg', 'media': {'format': 'audio', 'availability': 'Available to listen', 'expires': None}, 'position': None, 'has_clips': False, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p0232kg2'}, 'is_available_mediaset_pc_sd': True, 'has_segment_events': False, 'display_titles': {'title': 'The Tour de France', 'subtitle': ''}, 'has_medium_or_long_synopsis': True, 'has_related_links': False, 'media_type': 'audio', 'title': 'The Tour de France', 'is_legacy_media': False, 'type': 'episode', 'short_synopsis': "What is the best body type to win the Tour de France's yellow jersey?", 'actual_start': '2014-07-18T20:01:58+01:00', 'first_broadcast_date': '2014-07-18T19:50:00+01:00', 'duration': 660}}, {'programme': {'programme': {'pid': 'p00msxfl', 'title': 'More or Less', 'position': None, 'type': 'brand', 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01cnq9j'}, 'first_broadcast_date': '2012-01-13T23:50:00Z', 'expected_child_count': None}, 'pid': 'p0224gwp', 'media': {'format': 'audio', 'availability': 'Available to listen', 'expires': None}, 'position': None, 'has_clips': False, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p022k531'}, 'is_available_mediaset_pc_sd': True, 'has_segment_events': False, 'display_titles': {'title': 'Golden Ticket', 'subtitle': ''}, 'has_medium_or_long_synopsis': True, 'has_related_links': False, 'media_type': 'audio', 'title': 'Golden Ticket', 'is_legacy_media': False, 'type': 'episode', 'short_synopsis': "What are Charlie's odds of finding a golden ticket in Charlie and the Chocolate Factory?", 'actual_start': '2014-07-11T20:03:15+01:00', 'first_broadcast_date': '2014-07-11T19:50:00+01:00', 'duration': 660}}, {'programme': {'programme': {'pid': 'p00msxfl', 'title': 'More or Less', 'position': None, 'type': 'brand', 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01cnq9j'}, 'first_broadcast_date': '2012-01-13T23:50:00Z', 'expected_child_count': None}, 'pid': 'p021q7h2', 'media': {'format': 'audio', 'availability': 'Available to listen', 'expires': None}, 'position': None, 'has_clips': False, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p0224rht'}, 'is_available_mediaset_pc_sd': True, 'has_segment_events': False, 'display_titles': {'title': 'Will we Die Before our Parents?', 'subtitle': ''}, 'has_medium_or_long_synopsis': True, 'has_related_links': False, 'media_type': 'audio', 'title': 'Will we Die Before our Parents?', 'is_legacy_media': False, 'type': 'episode', 'short_synopsis': 'Will the current generation have a shorter life span than their parents due to obesity?', 'actual_start': '2014-07-04T20:01:45+01:00', 'first_broadcast_date': '2014-07-04T19:50:00+01:00', 'duration': 660}}, {'programme': {'programme': {'pid': 'p00msxfl', 'title': 'More or Less', 'position': None, 'type': 'brand', 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01cnq9j'}, 'first_broadcast_date': '2012-01-13T23:50:00Z', 'expected_child_count': None}, 'pid': 'p021f1bd', 'media': {'format': 'audio', 'availability': 'Available to listen', 'expires': None}, 'position': None, 'has_clips': False, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p021lwhx'}, 'is_available_mediaset_pc_sd': True, 'has_segment_events': False, 'display_titles': {'title': 'Is this the greatest World Cup ever?', 'subtitle': ''}, 'has_medium_or_long_synopsis': True, 'has_related_links': False, 'media_type': 'audio', 'title': 'Is this the greatest World Cup ever?', 'is_legacy_media': False, 'type': 'episode', 'short_synopsis': 'We cast a sceptical eye over the statistics. Plus: are we seeing the death of tiki-taka?', 'actual_start': '2014-06-30T00:01:55+01:00', 'first_broadcast_date': '2014-06-29T23:50:00+01:00', 'duration': 660}}, {'programme': {'programme': {'pid': 'p00msxfl', 'title': 'More or Less', 'position': None, 'type': 'brand', 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01cnq9j'}, 'first_broadcast_date': '2012-01-13T23:50:00Z', 'expected_child_count': None}, 'pid': 'p020wmlx', 'media': {'format': 'audio', 'availability': 'Available to listen', 'expires': None}, 'position': None, 'has_clips': False, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p021689j'}, 'is_available_mediaset_pc_sd': True, 'has_segment_events': False, 'display_titles': {'title': 'Money for Nothing?', 'subtitle': ''}, 'has_medium_or_long_synopsis': True, 'has_related_links': False, 'media_type': 'audio', 'title': 'Money for Nothing?', 'is_legacy_media': False, 'type': 'episode', 'short_synopsis': 'When it comes to aid, what works best - providing goods, or handing over cash?', 'actual_start': '2014-06-20T20:01:53+01:00', 'first_broadcast_date': '2014-06-20T19:50:00+01:00', 'duration': 660}}, {'programme': {'programme': {'pid': 'p00msxfl', 'title': 'More or Less', 'position': None, 'type': 'brand', 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01cnq9j'}, 'first_broadcast_date': '2012-01-13T23:50:00Z', 'expected_child_count': None}, 'pid': 'p020gkp9', 'media': {'format': 'audio', 'availability': 'Available to listen', 'expires': None}, 'position': None, 'has_clips': False, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p020y70g'}, 'is_available_mediaset_pc_sd': True, 'has_segment_events': False, 'display_titles': {'title': 'Heads Or Tails?', 'subtitle': ''}, 'has_medium_or_long_synopsis': True, 'has_related_links': False, 'media_type': 'audio', 'title': 'Heads Or Tails?', 'is_legacy_media': False, 'type': 'episode', 'short_synopsis': 'What can we learn about happiness if people make key life decisions based on a coin toss?', 'actual_start': '2014-06-13T20:01:55+01:00', 'first_broadcast_date': '2014-06-13T19:50:00+01:00', 'duration': 660}}, {'programme': {'programme': {'pid': 'p00msxfl', 'title': 'More or Less', 'position': None, 'type': 'brand', 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01cnq9j'}, 'first_broadcast_date': '2012-01-13T23:50:00Z', 'expected_child_count': None}, 'pid': 'p0201hpg', 'media': {'format': 'audio', 'availability': 'Available to listen', 'expires': None}, 'position': None, 'has_clips': False, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p020hqx2'}, 'is_available_mediaset_pc_sd': True, 'has_segment_events': False, 'display_titles': {'title': 'Spurious Correlations', 'subtitle': ''}, 'has_medium_or_long_synopsis': True, 'has_related_links': False, 'media_type': 'audio', 'title': 'Spurious Correlations', 'is_legacy_media': False, 'type': 'episode', 'short_synopsis': 'Is the divorce rate in the US state of Maine linked to margarine consumption?', 'actual_start': '2014-06-06T20:01:50+01:00', 'first_broadcast_date': '2014-06-06T19:50:00+01:00', 'duration': 660}}, {'programme': {'programme': {'pid': 'p00msxfl', 'title': 'More or Less', 'position': None, 'type': 'brand', 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01cnq9j'}, 'first_broadcast_date': '2012-01-13T23:50:00Z', 'expected_child_count': None}, 'pid': 'p01zmghb', 'media': {'format': 'audio', 'availability': 'Available to listen', 'expires': None}, 'position': None, 'has_clips': False, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p0202b1t'}, 'is_available_mediaset_pc_sd': True, 'has_segment_events': False, 'display_titles': {'title': 'The Piketty Affair', 'subtitle': ''}, 'has_medium_or_long_synopsis': True, 'has_related_links': False, 'media_type': 'audio', 'title': 'The Piketty Affair', 'is_legacy_media': False, 'type': 'episode', 'short_synopsis': 'Did ‘rock-star’ French economist Thomas Piketty get his numbers wrong?', 'actual_start': '2014-05-30T20:01:58+01:00', 'first_broadcast_date': '2014-05-30T19:50:00+01:00', 'duration': 660}}, {'programme': {'programme': {'pid': 'p00msxfl', 'title': 'More or Less', 'position': None, 'type': 'brand', 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01cnq9j'}, 'first_broadcast_date': '2012-01-13T23:50:00Z', 'expected_child_count': None}, 'pid': 'p01z6f6p', 'media': {'format': 'audio', 'availability': 'Available to listen', 'expires': None}, 'position': None, 'has_clips': False, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p0201x02'}, 'is_available_mediaset_pc_sd': True, 'has_segment_events': False, 'display_titles': {'title': 'Risk Savvy', 'subtitle': ''}, 'has_medium_or_long_synopsis': True, 'has_related_links': False, 'media_type': 'audio', 'title': 'Risk Savvy', 'is_legacy_media': False, 'type': 'episode', 'short_synopsis': 'Gerd Gigerenzer on the famous probability puzzle involving goats and game shows', 'actual_start': '2014-05-23T20:01:54+01:00', 'first_broadcast_date': '2014-05-23T19:50:00+01:00', 'duration': 660}}, {'programme': {'programme': {'pid': 'p00msxfl', 'title': 'More or Less', 'position': None, 'type': 'brand', 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01cnq9j'}, 'first_broadcast_date': '2012-01-13T23:50:00Z', 'expected_child_count': None}, 'pid': 'p01ys3t5', 'media': {'format': 'audio', 'availability': 'Available to listen', 'expires': None}, 'position': None, 'has_clips': False, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01z7cwk'}, 'is_available_mediaset_pc_sd': True, 'has_segment_events': False, 'display_titles': {'title': 'Did Global Poverty Halve Overnight?', 'subtitle': ''}, 'has_medium_or_long_synopsis': True, 'has_related_links': False, 'media_type': 'audio', 'title': 'Did Global Poverty Halve Overnight?', 'is_legacy_media': False, 'type': 'episode', 'short_synopsis': 'Does greater purchasing power in developing countries mean there are fewer poor people?', 'actual_start': '2014-05-16T20:01:50+01:00', 'first_broadcast_date': '2014-05-16T19:50:00+01:00', 'duration': 660}}, {'programme': {'programme': {'pid': 'p00msxfl', 'title': 'More or Less', 'position': None, 'type': 'brand', 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01cnq9j'}, 'first_broadcast_date': '2012-01-13T23:50:00Z', 'expected_child_count': None}, 'pid': 'p01ycqg2', 'media': {'format': 'audio', 'availability': 'Available to listen', 'expires': None}, 'position': None, 'has_clips': False, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01yt7mh'}, 'is_available_mediaset_pc_sd': True, 'has_segment_events': False, 'display_titles': {'title': 'Brazil’s Maths Superstar', 'subtitle': ''}, 'has_medium_or_long_synopsis': True, 'has_related_links': False, 'media_type': 'audio', 'title': 'Brazil’s Maths Superstar', 'is_legacy_media': False, 'type': 'episode', 'short_synopsis': "Alex Bellos tells the surprising story of Brazil's favourite maths book.", 'actual_start': '2014-05-09T20:02:06+01:00', 'first_broadcast_date': '2014-05-09T19:50:00+01:00', 'duration': 660}}, {'programme': {'programme': {'pid': 'p00msxfl', 'title': 'More or Less', 'position': None, 'type': 'brand', 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01cnq9j'}, 'first_broadcast_date': '2012-01-13T23:50:00Z', 'expected_child_count': None}, 'pid': 'p01y27t9', 'media': {'format': 'audio', 'availability': 'Available to listen', 'expires': None}, 'position': None, 'has_clips': False, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01ybcxn'}, 'is_available_mediaset_pc_sd': True, 'has_segment_events': False, 'display_titles': {'title': 'Did Sir Roger Bannister make the ‘impossible’ possible?', 'subtitle': ''}, 'has_medium_or_long_synopsis': True, 'has_related_links': False, 'media_type': 'audio', 'title': 'Did Sir Roger Bannister make the ‘impossible’ possible?', 'is_legacy_media': False, 'type': 'episode', 'short_synopsis': 'A mile in under four minutes. Did positive thinking propel dozens to do the same?', 'actual_start': '2014-05-05T00:01:54+01:00', 'first_broadcast_date': '2014-05-04T23:50:00+01:00', 'duration': 660}}, {'programme': {'programme': {'pid': 'p00msxfl', 'title': 'More or Less', 'position': None, 'type': 'brand', 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01cnq9j'}, 'first_broadcast_date': '2012-01-13T23:50:00Z', 'expected_child_count': None}, 'pid': 'p01xmswc', 'media': {'format': 'audio', 'availability': 'Available to listen', 'expires': None}, 'position': None, 'has_clips': False, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01xyp6w'}, 'is_available_mediaset_pc_sd': True, 'has_segment_events': False, 'display_titles': {'title': 'Killed for Being Female?', 'subtitle': ''}, 'has_medium_or_long_synopsis': True, 'has_related_links': False, 'media_type': 'audio', 'title': 'Killed for Being Female?', 'is_legacy_media': False, 'type': 'episode', 'short_synopsis': 'Are there 100 million women missing from the world?', 'actual_start': '2014-04-25T20:02:04+01:00', 'first_broadcast_date': '2014-04-25T19:50:00+01:00', 'duration': 660}}, {'programme': {'programme': {'pid': 'p00msxfl', 'title': 'More or Less', 'position': None, 'type': 'brand', 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01cnq9j'}, 'first_broadcast_date': '2012-01-13T23:50:00Z', 'expected_child_count': None}, 'pid': 'p01x7n87', 'media': {'format': 'audio', 'availability': 'Available to listen', 'expires': None}, 'position': None, 'has_clips': False, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01x8vgs'}, 'is_available_mediaset_pc_sd': True, 'has_segment_events': False, 'display_titles': {'title': 'Magic Numbers', 'subtitle': ''}, 'has_medium_or_long_synopsis': True, 'has_related_links': False, 'media_type': 'audio', 'title': 'Magic Numbers', 'is_legacy_media': False, 'type': 'episode', 'short_synopsis': 'Do you have a favourite number - one you think stands out from all the others?', 'actual_start': '2014-04-18T20:01:55+01:00', 'first_broadcast_date': '2014-04-18T19:50:00+01:00', 'duration': 660}}, {'programme': {'programme': {'pid': 'p00msxfl', 'title': 'More or Less', 'position': None, 'type': 'brand', 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01cnq9j'}, 'first_broadcast_date': '2012-01-13T23:50:00Z', 'expected_child_count': None}, 'pid': 'p01wtd1b', 'media': {'format': 'audio', 'availability': 'Available to listen', 'expires': None}, 'position': None, 'has_clips': False, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01x62qy'}, 'is_available_mediaset_pc_sd': True, 'has_segment_events': False, 'display_titles': {'title': 'Nigeria - Rich or Poor?', 'subtitle': ''}, 'has_medium_or_long_synopsis': True, 'has_related_links': False, 'media_type': 'audio', 'title': 'Nigeria - Rich or Poor?', 'is_legacy_media': False, 'type': 'episode', 'short_synopsis': 'How did Nigeria become the largest African economy overnight?', 'actual_start': '2014-04-11T20:02:47+01:00', 'first_broadcast_date': '2014-04-11T19:50:00+01:00', 'duration': 660}}, {'programme': {'programme': {'pid': 'p00msxfl', 'title': 'More or Less', 'position': None, 'type': 'brand', 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01cnq9j'}, 'first_broadcast_date': '2012-01-13T23:50:00Z', 'expected_child_count': None}, 'pid': 'p01wd03f', 'media': {'format': 'audio', 'availability': 'Available to listen', 'expires': None}, 'position': None, 'has_clips': False, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01wlr0p'}, 'is_available_mediaset_pc_sd': True, 'has_segment_events': False, 'display_titles': {'title': 'Freedom in Numbers', 'subtitle': ''}, 'has_medium_or_long_synopsis': True, 'has_related_links': False, 'media_type': 'audio', 'title': 'Freedom in Numbers', 'is_legacy_media': False, 'type': 'episode', 'short_synopsis': 'How many people in the world live in freedom? First broadcast live on 01 April 2014.', 'actual_start': '2014-04-07T00:01:58+01:00', 'first_broadcast_date': '2014-04-06T23:50:00+01:00', 'duration': 660}}, {'programme': {'programme': {'pid': 'p00msxfl', 'title': 'More or Less', 'position': None, 'type': 'brand', 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01cnq9j'}, 'first_broadcast_date': '2012-01-13T23:50:00Z', 'expected_child_count': None}, 'pid': 'p01w8qqm', 'media': {'format': 'audio', 'availability': 'Available to listen', 'expires': None}, 'position': None, 'has_clips': False, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01wlr0p'}, 'is_available_mediaset_pc_sd': True, 'has_segment_events': False, 'display_titles': {'title': 'Freedom in Numbers', 'subtitle': ''}, 'has_medium_or_long_synopsis': True, 'has_related_links': False, 'media_type': 'audio', 'title': 'Freedom in Numbers', 'is_legacy_media': False, 'type': 'episode', 'short_synopsis': 'How many people in the world live in freedom?', 'actual_start': '2014-04-01T09:01:48+01:00', 'first_broadcast_date': '2014-04-01T08:50:00+01:00', 'duration': 660}}, {'programme': {'programme': {'pid': 'p00msxfl', 'title': 'More or Less', 'position': None, 'type': 'brand', 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01cnq9j'}, 'first_broadcast_date': '2012-01-13T23:50:00Z', 'expected_child_count': None}, 'pid': 'p01w26gt', 'media': {'format': 'audio', 'availability': 'Available to listen', 'expires': None}, 'position': None, 'has_clips': False, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01wdqpf'}, 'is_available_mediaset_pc_sd': True, 'has_segment_events': False, 'display_titles': {'title': 'Is London France’s Sixth Largest City?', 'subtitle': ''}, 'has_medium_or_long_synopsis': True, 'has_related_links': False, 'media_type': 'audio', 'title': 'Is London France’s Sixth Largest City?', 'is_legacy_media': False, 'type': 'episode', 'short_synopsis': 'Are there really 300,000 French people in London?', 'actual_start': '2014-03-29T12:41:30Z', 'first_broadcast_date': '2014-03-29T11:50:00Z', 'duration': 660}}, {'programme': {'programme': {'pid': 'p00msxfl', 'title': 'More or Less', 'position': None, 'type': 'brand', 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01cnq9j'}, 'first_broadcast_date': '2012-01-13T23:50:00Z', 'expected_child_count': None}, 'pid': 'p01v5sqh', 'media': {'format': 'audio', 'availability': 'Available to listen', 'expires': None}, 'position': None, 'has_clips': False, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01w07qh'}, 'is_available_mediaset_pc_sd': True, 'has_segment_events': False, 'display_titles': {'title': 'Missing Planes', 'subtitle': ''}, 'has_medium_or_long_synopsis': True, 'has_related_links': False, 'media_type': 'audio', 'title': 'Missing Planes', 'is_legacy_media': False, 'type': 'episode', 'short_synopsis': 'Could Bayesian statistics help find Flight MH370?', 'actual_start': '2014-03-22T12:30:51Z', 'first_broadcast_date': '2014-03-22T11:50:00Z', 'duration': 660}}, {'programme': {'programme': {'pid': 'p00msxfl', 'title': 'More or Less', 'position': None, 'type': 'brand', 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01cnq9j'}, 'first_broadcast_date': '2012-01-13T23:50:00Z', 'expected_child_count': None}, 'pid': 'p01tjcwp', 'media': {'format': 'audio', 'availability': 'Available to listen', 'expires': None}, 'position': None, 'has_clips': False, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01v34fc'}, 'is_available_mediaset_pc_sd': True, 'has_segment_events': True, 'display_titles': {'title': 'Mailbox Edition', 'subtitle': ''}, 'has_medium_or_long_synopsis': True, 'has_related_links': False, 'media_type': 'audio', 'title': 'Mailbox Edition', 'is_legacy_media': False, 'type': 'episode', 'short_synopsis': 'Your questions – from the USA, Australia and the world', 'actual_start': '2014-03-15T12:34:36Z', 'segments_title': 'Chapters', 'first_broadcast_date': '2014-03-15T11:50:00Z', 'duration': 660}}, {'programme': {'programme': {'pid': 'p00msxfl', 'title': 'More or Less', 'position': None, 'type': 'brand', 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01cnq9j'}, 'first_broadcast_date': '2012-01-13T23:50:00Z', 'expected_child_count': None}, 'pid': 'p01t399q', 'media': {'format': 'audio', 'availability': 'Available to listen', 'expires': None}, 'position': None, 'has_clips': False, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01tgssx'}, 'is_available_mediaset_pc_sd': True, 'has_segment_events': False, 'display_titles': {'title': 'Modern Slavery', 'subtitle': ''}, 'has_medium_or_long_synopsis': True, 'has_related_links': False, 'media_type': 'audio', 'title': 'Modern Slavery', 'is_legacy_media': False, 'type': 'episode', 'short_synopsis': 'Are there 21 million slaves in the world today?', 'actual_start': '2014-03-08T12:43:36Z', 'first_broadcast_date': '2014-03-08T11:50:00Z', 'duration': 660}}, {'programme': {'programme': {'pid': 'p00msxfl', 'title': 'More or Less', 'position': None, 'type': 'brand', 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01cnq9j'}, 'first_broadcast_date': '2012-01-13T23:50:00Z', 'expected_child_count': None}, 'pid': 'p01sqly1', 'media': {'format': 'audio', 'availability': 'Available to listen', 'expires': None}, 'position': None, 'has_clips': False, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01t1h8z'}, 'is_available_mediaset_pc_sd': True, 'has_segment_events': False, 'display_titles': {'title': 'The 10,000 Hours Rule', 'subtitle': ''}, 'has_medium_or_long_synopsis': True, 'has_related_links': False, 'media_type': 'audio', 'title': 'The 10,000 Hours Rule', 'is_legacy_media': False, 'type': 'episode', 'short_synopsis': 'Is it possible to become an expert on practice alone?', 'actual_start': '2014-03-01T12:46:59Z', 'first_broadcast_date': '2014-03-01T11:50:00Z', 'duration': 660}}, {'programme': {'programme': {'pid': 'p00msxfl', 'title': 'More or Less', 'position': None, 'type': 'brand', 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01cnq9j'}, 'first_broadcast_date': '2012-01-13T23:50:00Z', 'expected_child_count': None}, 'pid': 'p01sbmjs', 'media': {'format': 'audio', 'availability': 'Available to listen', 'expires': None}, 'position': None, 'has_clips': False, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01snvpn'}, 'is_available_mediaset_pc_sd': True, 'has_segment_events': True, 'display_titles': {'title': 'Neknomination Outbreak', 'subtitle': ''}, 'has_medium_or_long_synopsis': True, 'has_related_links': False, 'media_type': 'audio', 'title': 'Neknomination Outbreak', 'is_legacy_media': False, 'type': 'episode', 'short_synopsis': 'How quickly will the online global drinking craze Neknomination spread - and fizzle out?', 'actual_start': '2014-02-22T13:55:03Z', 'segments_title': 'Chapters', 'first_broadcast_date': '2014-02-22T11:50:00Z', 'duration': 660}}, {'programme': {'programme': {'pid': 'p00msxfl', 'title': 'More or Less', 'position': None, 'type': 'brand', 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01cnq9j'}, 'first_broadcast_date': '2012-01-13T23:50:00Z', 'expected_child_count': None}, 'pid': 'p01ryk69', 'media': {'format': 'audio', 'availability': 'Available to listen', 'expires': None}, 'position': None, 'has_clips': False, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01s8trv'}, 'is_available_mediaset_pc_sd': True, 'has_segment_events': False, 'display_titles': {'title': 'Love by Numbers', 'subtitle': ''}, 'has_medium_or_long_synopsis': True, 'has_related_links': False, 'media_type': 'audio', 'title': 'Love by Numbers', 'is_legacy_media': False, 'type': 'episode', 'short_synopsis': 'The maths behind modern matchmaking', 'actual_start': '2014-02-15T12:41:24Z', 'first_broadcast_date': '2014-02-15T11:50:00Z', 'duration': 660}}, {'programme': {'programme': {'pid': 'p00msxfl', 'title': 'More or Less', 'position': None, 'type': 'brand', 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01cnq9j'}, 'first_broadcast_date': '2012-01-13T23:50:00Z', 'expected_child_count': None}, 'pid': 'p01r89yx', 'media': {'format': 'audio', 'availability': 'Available to listen', 'expires': None}, 'position': None, 'has_clips': False, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01rxdvc'}, 'is_available_mediaset_pc_sd': True, 'has_segment_events': False, 'display_titles': {'title': 'Rising Drug Overdose Deaths', 'subtitle': ''}, 'has_medium_or_long_synopsis': True, 'has_related_links': False, 'media_type': 'audio', 'title': 'Rising Drug Overdose Deaths', 'is_legacy_media': False, 'type': 'episode', 'short_synopsis': 'Deaths from drug overdoses in the United States are on the rise – but why?', 'actual_start': '2014-02-08T12:40:21Z', 'first_broadcast_date': '2014-02-08T11:50:00Z', 'duration': 660}}, {'programme': {'programme': {'pid': 'p00msxfl', 'title': 'More or Less', 'position': None, 'type': 'brand', 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01cnq9j'}, 'first_broadcast_date': '2012-01-13T23:50:00Z', 'expected_child_count': None}, 'pid': 'p01qkfnp', 'media': {'format': 'audio', 'availability': 'Available to listen', 'expires': None}, 'position': None, 'has_clips': False, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01r5k8b'}, 'is_available_mediaset_pc_sd': True, 'has_segment_events': False, 'display_titles': {'title': 'Immigration', 'subtitle': ''}, 'has_medium_or_long_synopsis': True, 'has_related_links': True, 'media_type': 'audio', 'title': 'Immigration', 'is_legacy_media': False, 'type': 'episode', 'short_synopsis': 'Tim Harford explores the economic benefits of immigration, plus wedding guest strategies.', 'actual_start': '2014-02-01T12:49:18Z', 'first_broadcast_date': '2014-02-01T11:50:00Z', 'duration': 660}}, {'programme': {'programme': {'pid': 'p00msxfl', 'title': 'More or Less', 'position': None, 'type': 'brand', 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01cnq9j'}, 'first_broadcast_date': '2012-01-13T23:50:00Z', 'expected_child_count': None}, 'pid': 'p01q5cfc', 'media': {'format': 'audio', 'availability': 'Available to listen', 'expires': None}, 'position': None, 'has_clips': False, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01qj37z'}, 'is_available_mediaset_pc_sd': True, 'has_segment_events': False, 'display_titles': {'title': 'Alcohol and Cancer', 'subtitle': ''}, 'has_medium_or_long_synopsis': True, 'has_related_links': False, 'media_type': 'audio', 'title': 'Alcohol and Cancer', 'is_legacy_media': False, 'type': 'episode', 'short_synopsis': 'Do two large glasses of wine triple the risk of mouth cancer?', 'actual_start': '2014-01-25T12:30:13Z', 'first_broadcast_date': '2014-01-25T11:50:00Z', 'duration': 660}}, {'programme': {'programme': {'pid': 'p00msxfl', 'title': 'More or Less', 'position': None, 'type': 'brand', 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01cnq9j'}, 'first_broadcast_date': '2012-01-13T23:50:00Z', 'expected_child_count': None}, 'pid': 'p01pnzhr', 'media': {'format': 'audio', 'availability': 'Available to listen', 'expires': None}, 'position': None, 'has_clips': False, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01q3vpc'}, 'is_available_mediaset_pc_sd': True, 'has_segment_events': False, 'display_titles': {'title': 'Could an Apple-a-day Reduce Illness?', 'subtitle': ''}, 'has_medium_or_long_synopsis': True, 'has_related_links': False, 'media_type': 'audio', 'title': 'Could an Apple-a-day Reduce Illness?', 'is_legacy_media': False, 'type': 'episode', 'short_synopsis': 'A study has shown an apple-a-day will actually keep the doctors away but is it accurate?', 'actual_start': '2014-01-18T12:29:49Z', 'first_broadcast_date': '2014-01-18T11:50:00Z', 'duration': 660}}, {'programme': {'programme': {'pid': 'p00msxfl', 'title': 'More or Less', 'position': None, 'type': 'brand', 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01cnq9j'}, 'first_broadcast_date': '2012-01-13T23:50:00Z', 'expected_child_count': None}, 'pid': 'p01p1742', 'media': {'format': 'audio', 'availability': 'Available to listen', 'expires': None}, 'position': None, 'has_clips': False, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'image': {'pid': 'p01pn34n'}, 'is_available_mediaset_pc_sd': True, 'has_segment_events': False, 'display_titles': {'title': 'Counting the Dead in Iraq', 'subtitle': ''}, 'has_medium_or_long_synopsis': True, 'has_related_links': False, 'media_type': 'audio', 'title': 'Counting the Dead in Iraq', 'is_legacy_media': False, 'type': 'episode', 'short_synopsis': 'How do you count those killed in war accurately?', 'actual_start': '2014-01-11T12:30:06Z', 'first_broadcast_date': '2014-01-11T11:50:00Z', 'duration': 660}}], 'offset': 0, 'page': 1}
#Test the episode URL creator
for episode in data['episodes'][:3]:
print(programmeURL(episode['programme']['pid']))
http://www.bbc.co.uk/programmes/p023276m.json http://www.bbc.co.uk/programmes/p022m7bg.json http://www.bbc.co.uk/programmes/p0224gwp.json
#Test that we can get the full data for episode from a created URL
episodeData=getData(programmeURL(episode['programme']['pid']))
episodeData
{'programme': {'supporting_content_items': [], 'pid': 'p0224gwp', 'position': None, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'type': 'episode', 'display_title': {'title': 'More or Less', 'subtitle': 'Golden Ticket'}, 'categories': [{'broader': {}, 'title': 'Factual', 'sameAs': None, 'type': 'genre', 'key': 'factual', 'id': 'C00045', 'narrower': [], 'has_topic_page': False}, {'broader': {}, 'title': 'Magazines & Reviews', 'sameAs': None, 'type': 'format', 'key': 'magazinesandreviews', 'id': 'PT009', 'narrower': [], 'has_topic_page': False}], 'parent': {'programme': {'pid': 'p00msxfl', 'title': 'More or Less', 'position': None, 'image': {'pid': 'p01cnq9j'}, 'ownership': {'service': {'type': 'radio', 'key': 'worldserviceradio', 'id': 'bbc_world_service', 'title': 'BBC World Service'}}, 'type': 'brand', 'short_synopsis': 'Tim Harford explains the numbers and statistics in the news and in life.', 'expected_child_count': None, 'first_broadcast_date': '2012-01-13T23:50:00Z'}}, 'links': [], 'long_synopsis': 'In the film and musical based on Roald Dahl’s novel Charlie and the Chocolate Factory, Charlie Bucket receives a golden ticket to visit Willy Wonka’s chocolate factory. But one of our younger More or Less listeners in England wanted to find out what the chances would be of winning one of those Golden Tickets. So we sent maths book author Rob Eastaway to her school in Derby to explain the answer to her class-mates. A must-listen for anyone who struggles to get their head around probability. \r\n\r\nAlso on the programme we look at whether the age of players makes a difference in World Cup football. \r\n\r\n(Photo: A Wonka bar. Credit: Getty Images)', 'media_type': 'audio', 'title': 'Golden Ticket', 'peers': {'previous': {'pid': 'p021q7h2', 'media_type': 'audio', 'title': 'Will we Die Before our Parents?', 'position': None, 'type': 'episode', 'first_broadcast_date': '2014-07-04T19:50:00+01:00'}, 'next': {'pid': 'p022m7bg', 'media_type': 'audio', 'title': 'The Tour de France', 'position': None, 'type': 'episode', 'first_broadcast_date': '2014-07-18T19:50:00+01:00'}}, 'image': {'pid': 'p022k531'}, 'versions': [{'canonical': 1, 'pid': 'p0224gys', 'duration': 660, 'types': ['Original version']}], 'short_synopsis': "What are Charlie's odds of finding a golden ticket in Charlie and the Chocolate Factory?", 'expected_child_count': None, 'first_broadcast_date': '2014-07-11T19:50:00+01:00', 'medium_synopsis': "In Charlie and the Chocolate Factory, Charlie Bucket wins a golden ticket to visit Willy Wonka’s factory. What were his odds? Plus: World Cup football and players' ages."}}
#Long_synopsis data is often good to search on
episodeData['programme']['long_synopsis']
'In the film and musical based on Roald Dahl’s novel Charlie and the Chocolate Factory, Charlie Bucket receives a golden ticket to visit Willy Wonka’s chocolate factory. But one of our younger More or Less listeners in England wanted to find out what the chances would be of winning one of those Golden Tickets. So we sent maths book author Rob Eastaway to her school in Derby to explain the answer to her class-mates. A must-listen for anyone who struggles to get their head around probability. \r\n\r\nAlso on the programme we look at whether the age of players makes a difference in World Cup football. \r\n\r\n(Photo: A Wonka bar. Credit: Getty Images)'
#We're going to store the data in a MongoDB - so let's set one up
from pymongo import MongoClient
c = MongoClient('localhost', 27017)
db = c['bbc-database']
collection = db.more_or_less
#Test saving the data
#Let's use the episode ID as the MongoDB key _id
episodeData['programme']['_id']=episodeData['programme']['pid']
#Use save rather than insert (insert throws an error if by chance there is a duplicate key _id)
collection.save(episodeData['programme'])
'p0224gwp'
#Test a search
for r in collection.find({'long_synopsis': {'$regex':'deaths'}},{'long_synopsis':1}):
print(r)
#Now we're going to put all the pieces together
seriesStub='http://www.bbc.co.uk/programmes/b006qshd/episodes/player.json?page=' #More or Less, Radio 4
#The scraping flag just helps us keep track of the pages we're going to iterate through
scraping=True
pagecount=1
while scraping:
url=seriesStub+str(pagecount)
resp = requests.get(url)
data=json.loads(resp.content.decode('utf8'))
if data['page']!=pagecount: scraping=False
else:
for episode in data['episodes']:
episodeData=getData(programmeURL(episode['programme']['pid']))
episodeData['programme']['_id']=episodeData['programme']['pid']
collection.save(episodeData['programme'])
pagecount+=1
collection.count()
107
Let's abstract a little more:
seriesStubGenerator=lambda pid: 'http://www.bbc.co.uk/programmes/'+pid+'/episodes/player.json?page='
def bbcSeriesDataScraper(collection,pid):
scraping=True
pagecount=1
seriesStub=seriesStubGenerator(pid)
while scraping:
url=seriesStub+str(pagecount)
resp = requests.get(url)
data=json.loads(resp.content.decode('utf8'))
if data['page']!=pagecount: scraping=False
else:
for episode in data['episodes']:
episodeData=getData(programmeURL(episode['programme']['pid']))
episodeData['programme']['_id']=episodeData['programme']['pid']
collection.save(episodeData['programme'])
pagecount+=1
#seriesStub='http://www.bbc.co.uk/programmes/p00msxfl/episodes/player.json?page=' #More or Less, World Service
bbcSeriesDataScraper(collection,'p00msxfl')
collection.count()
240
#Test a search
for r in collection.find({'long_synopsis': {'$regex':'deaths'}},{'long_synopsis':1}):
print(r)
{'long_synopsis': 'Tim Harford tries to find out why this is the week of the year in which most deaths occur. It\'s often thought that the spike in so-called \'excess\' winter deaths is caused at least in part by the fact that some people living in poverty can\'t afford to turn up the central heating when freezing weather sets in. But Tim discovers the picture is much more complicated than that. He goes through the data with Claudia Wells from the Office for National Statistics, and Professor Rachel Vreeman of the Indiana University School of Medicine in the United States.\n\nAre there really two million millionaire pensioners in the UK? The Intergenerational Foundation says two million over 60-year-olds are living in "millionaire households" and, it argues, don\'t need their pensions protected by the government. Tim looks into the numbers.\n\nEstimates of the death count in Iraq since the start of the war in 2003 range from 100,000 to one million. Tim explores why such a range exists and what methods are used to count those killed during war. And he discovers why the death count hasn\'t stopped Iraq\'s population growing by almost a third in that time. He speaks to Glen Rangwala from Cambridge University and Patrick Gerland from the UN\'s demographics team.\n\nAnd after last week\'s abject apology from mathematician and comedian, Matt Parker, he is back to apologise for that apology. This time Tim makes him answer for the unforgiveable comments he made about the nautical mile.\n\nPresenter: Tim Harford\nProducer: Ruth Alexander.', '_id': 'b03nt8bf'} {'long_synopsis': "As global leaders remain divided on whether to carry out a military strike against Syria in response to the apparent use of chemical weapons against its people, Tim Harford looks at the different claims made about how many people have been killed. The United States, the UK and France are sharing intelligence, but all quote different estimates of how many people they think died in the attack by Syrian President Bashar al-Assad's forces. Tim speaks to Kelly Greenhill, a professor of political science at Tufts University in the US, and co-author of Sex, Drugs and Body Counts about why the numbers vary so widely. And he speaks to Megan Price from the Human Rights Data Analysis Group who has been trying to keep a tally of the deaths in Syria since the conflict began.\n\nThe cost of care has forced a million families to sell their homes in the past five years, according to the Daily Telegraph. It's quoting research commissioned by NFU Mutual and carried out by ICM. But Tim Harford spots some tell-tale signs that the survey respondents may not all have been telling the truth.\nWhat can statistics tell us about the safety of Super Puma helicopters, used by the offshore oil and gas industry? Tim Harford looks at the numbers, following a fatal accident off Shetland in August - the fifth incident involving Super Pumas in the North Sea since 2009.\n\nApparently, it's a fact that if there's one thing that's worse for you than drinking, scoffing bacon sandwiches and smoking 80 unfiltered cigarettes a day, it's being left-handed. Left-handers die on average several years earlier than right-handers. Or do they? Tim gets to the bottom of a sinister statistic with Professor Chris McManus, author of Right Hand, Left Hand.\n\nMore than 300,000 attempts were made to access pornographic websites at the Houses of Parliament in the past year, official records suggest. But with 15 attempts made in one month and almost 115,000 in another, the figures themselves raised an eyebrow at More or Less HQ - they just don't make sense. Tim speaks to Fergus Reid from Parliament's ICT team.\n\nAnd finally, was Labour MP Fiona Mactaggart right to calculate that Britons have spent 76 centuries hanging on the phone to get through to government departments in just one year? She checks her sums.\n\nPresenter: Tim Harford\nProducer: Ruth Alexander.", '_id': 'b039dbk8'} {'long_synopsis': 'Has the government taken into account the worth of a badger\'s life in any cost-benefit analysis of the controversial badger cull, which is taking place to tackle the spread of tuberculosis among cattle? Tim Harford considers the problem. And the government aims to kill 70% of badgers in the two cull zones, but Tim discovers that such precision might be tricky. It\'s terribly difficult to count badgers, you see.\n\nPlus, have blundering doctors and nurses really killed 13,000 people? This was widely reported alongside the publication of the Keogh Report into standards of care at 14 NHS hospital trusts in England. Tim Harford finds out how so-called \'excess\' deaths are calculated, and whether they\'re the best measure of hospital standards.\n\nThe shadow immigration minister Chris Bryant has warned that climate change is going to create 200 million more migrants. But More or Less discovers that migration experts disagree. \n\nAnd, always down with the cool kids, Tim discovers more about this buzz phrase, "big data". Companies and governments are releasing large datasets about us, with our identities obscured, for the purposes of marketing - or even, occasionally, for the purposes of public understanding. But might those apparently anonymous datasets be telling the world our darkest secrets?\n\nPresenter: Tim Harford\nProducer: Ruth Alexander.', '_id': 'b038zhb7'} {'long_synopsis': "The UK Independence Party (UKIP) made substantial gains in recent local elections.\nIt's best known for wanting the UK to withdraw from the European Union. This is not an official policy shared by any of the other established political parties.\nAnd, as with any political battleground, numbers have been pressed into service as weapons.\nTim Harford makes sense of the numbers flying around with the help of Iain Begg, professorial research fellow at the London School of Economics' European Institute.\n\nMilitary suicides:\n\nIt's claimed that more Falklands veterans have taken their own lives than died during the conflict itself. But is it true? Next week the government's defence statistics agency will publish a long-awaited report about the number of military personnel who have killed themselves since serving in that conflict in the 1980s. More or Less reporter Charlotte McDonald speaks to Simon Wessely Director, King's Centre for Military Health Research Institute of Psychiatry about what estimates from Vietnam and the first Gulf War tell us about the mental health of war veterans - and about why the rate of deaths tells us more than the raw numbers do.\n\nWhy you really might be Richard III's relative:\n\nFifteen relatives of Richard III are petitioning the High Court about where the king should be buried. Some reporting has implied that the famous 15 are almost the only descendants of Richard III who exist. But mathematician Rob Eastaway figures out how many other distant relatives of Richard III might actually be out there.\n\n\nDog years:\n\nIt's often said that the age of dogs can be better understood by multiplying their age, in human years, by seven. But is that really true? Ben Carter invents the More or Less Dogulator. Use it to calculate how old your dog is in human terms.\n\nPresenter: Tim Harford\nProducer: Ruth Alexander.", '_id': 'b01s8vvv'} {'long_synopsis': "Levelling the statistical playing field\n\nIf you adjust for the fact that some countries are richer than others, and some have more people in them, can we work out what the Olympic medal tally should look like, based only on those factors?\n\nGun control\n\nLast week's mass-shooting at a cinema in Colorado has - not surprisingly - intensified America's bitter and long-running argument with itself about gun control. The argument is political and highly partisan. But it is also practical: would tighter gun laws actually lead to fewer gun deaths? You might think it's obvious that they would. But it seems the evidence isn't quite that clear.\n\nTax\n\nThe treasury minister David Gauke came in for some stick this week for arguing that people who pay plumbers and cleaners cash-in-hand, while not breaking the law, are immoral. Several commentators have argued that the problem is small beer compared to the huge amounts sheltered from the taxman by large companies and rich individuals. Are they right?\n\nLeaders' mums\n\nListener Mike Shearing wrote to us after noticing that the mums of post-war US presidents seem to have died very late, while British prime ministerial mothers seem to die young. Had he - he asked - found something of significance? He certainly had.\n\nHow has Britain changed since 1908?\n\nA new book by researchers at the House of Commons Library charts in numbers how Britain has changed since it hosted the 1908 Olympics. Their findings may surprise you.\n\nPresenter: Tim Harford\nProducer: Richard Knight.", '_id': 'b01l1g64'} {'long_synopsis': 'Gay Britain\n\nAccording to recent Office for National Statistics figures the UK has fewer gay, lesbian and bisexual people living in it than we thought. But gay rights groups have questioned the number. Why is it so much lower than previous estimates? More or Less investigates.\n\nPaying for bankers\' mistakes\n\nThe Irish Central Bank has released an estimate of the total cost of bailing out its banking sector: about 45bn euros, or £39bn. We ask a favourite More or Less question: is that really a big number?\n\n"Needless" deaths?\n\n"More than 70 children may have died needlessly after heart surgery" claimed a recent Sunday Telegraph headline. But the story was based on research led by David Spiegelhalter, a medical statistician and no stranger to More or Less, who thinks the reporter misunderstood his numbers.\n\nThe PM \'s pay\n\nLast week we noted that the prime minister\'s salary has become a convenient benchmark against which other public sector workers\' pay is measured. But you wanted us to go further and work out the PM\'s total reward. So we have.\n\nThe micromort\n\nWe examine the micromort, a cheeky little unit of risk, with Matt Parker, the "stand up mathematician". Matt\'s show at the Edinburgh Fringe was a raging success (which is what we\'d expect of a show about maths).\n\nPolice numbers\n\nAn HMIC report in July claimed that only 11% of police officers are available to respond to incidents at any one time. Since then the number - generally summarised as 1 in 10 - has been widely quoted. The implication, it seems, is that while one officer is fighting crime, the other 9 are holed up in the police station eating donuts or form-filling. But with the help of listener Tim Treffry we examine whether that\'s really the conclusion we should draw.', '_id': 'b00tznbk'} {'long_synopsis': "Which would win in a fight - a shark or a toaster? Tim Harford finds out in this week's More or Less. The team also investigate whether Hospital Standardised Mortality Ratios (or HSMRs) - expected deaths to observed deaths - can be unhelpful, ask who stands to lose from the scrapping of Child Trust Funds and remember the great mathematician, Martin Gardner.", '_id': 'b00sk7rm'} {'long_synopsis': "Which would win in a fight - a shark or a toaster? Tim Harford finds out in this week's More or Less. The team also investigate whether Hospital Standardised Mortality Ratios (or HSMRs) - expected deaths to observed deaths - can be unhelpful, ask who stands to lose from the scrapping of Child Trust Funds and remember the great mathematician, Martin Gardner.", '_id': 'b00sgbr5'} {'long_synopsis': 'Philip Seymour Hoffman has become the latest celebrity to die from taking drugs. His premature death made headlines around the world - but what is less reported is the rising number of deaths from drug overdoses of both illegal and prescription drugs in the US – outstripping those of road traffic accidents and firearm deaths. The team takes a look at the numbers.\r\nWe hear from Dr. Len Paulozzi from the Centres for Disease Control and Prevention and Isabelle Giraudon from the European Monitoring Centre for Drugs and Drug Addiction.', '_id': 'p01r89yx'} {'long_synopsis': "As global leaders remain divided on whether to carry out a military strike against Syria in response to the apparent use of chemical weapons against its people, Tim Harford looks at the different claims made about how many people have been killed. The United States, the UK and France are sharing intelligence, but all quote different estimates of how many people they think died in the attack by Syrian President Bashar al-Assad's forces. Tim speaks to Kelly Greenhill, a professor of political science at Tufts University in the US, and co-author of Sex, Drugs and Body Counts about why the numbers vary so widely. And he speaks to Megan Price from the Human Rights Data Analysis Group, who has been trying to keep a tally of the deaths in Syria since the conflict began.\r\n\r\nApparently, it is a fact that if there's one thing that's worse for you than drinking, scoffing bacon sandwiches and smoking 80 unfiltered cigarettes a day, it is being left-handed. Left-handers die on average several years earlier than right-handers. Or do they? Tim gets to the bottom of a sinister statistic with Professor Chris McManus, author of Right Hand, Left Hand.\r\n\r\n(Image: A man holds his forehead as he stands amongst rubble in Syria. Credit: AFP/Getty Images)", '_id': 'p01fyvkc'} {'long_synopsis': 'The recent massacre at a school in the United States has re-opened the debate on gun ownership in the United States. Tim Harford investigates whether the statistics on firearms deaths being widely shared on the internet stand up to scrutiny.\r\nAnd he discovers the mathematics of juggling, with Colin Wright.\r\nProducer: Ruth Alexander', '_id': 'p012184b'} {'long_synopsis': "Last week's mass-shooting at a cinema in Colorado has - not surprisingly - intensified America's bitter and long-running argument with itself about gun control. \n\nThe argument is political and highly partisan. But it is also practical: would tighter gun laws actually lead to fewer gun deaths? You might think it's obvious that they would. But it seems the evidence isn't quite that clear. \n\nAlso: how have Olympians changed in the last century?\n\n(Image: Shell casing for .40 caliber cartridges. Credit: Getty Images)", '_id': 'p00vtkct'}