Learning NLTK¶
This notebook introduces the Natural Language Toolkit (NLTK).
NLTK Example Texts¶
The first step is to import the nltk library and to load some example texts.
In [1]:
import nltk
In [2]:
from nltk.book import *
Dispersion Plots¶
In [3]:
text1.dispersion_plot(['Ahab','whale','Ishmael','Queequeg', 'Moby', 'dive'])
In [4]:
text4.dispersion_plot(["citizens", "democracy", "freedom", "duties", "America"])
Concordances¶
In [5]:
text1.concordance('Ishmael')
In [6]:
text1.similar('Ishmael')
In [7]:
text1.similar('whale')
In [8]:
text1.collocations()
In [9]:
text1.concordance("monstrous")
In [10]:
text2.concordance("monstrous")
In [11]:
text1.similar("monstrous")
In [12]:
text2.similar("monstrous")
Lexical Diversity¶
In [13]:
def lexical_diversity(text):
return len(text)/(1.0*len(set(text)))
def percentage(count, total):
return 100*count/total
In [14]:
lexical_diversity(text3)
Out[14]:
In [15]:
lexical_diversity(text4)
Out[15]:
In [16]:
lexical_diversity(text5)
Out[16]:
In [17]:
fdist1 = FreqDist(text1)
In [18]:
vocabulary1 = fdist1.keys()
In [19]:
fdist1['whale']
Out[19]:
In [20]:
fdist1['monstrous']
Out[20]:
In [21]:
fdist1.plot(50, cumulative=True)
In [22]:
hapaxes1 = fdist1.hapaxes()
print len(hapaxes1)
print hapaxes1[1000:1010]
n-grams¶
In [23]:
thursday_sents = nltk.corpus.gutenberg.sents('chesterton-thursday.txt')
sent22 = thursday_sents[22]
' '.join(sent22)
Out[23]:
In [24]:
nltk.bigrams(w for w in sent22 if w.isalpha())
Out[24]:
In [25]:
import networkx as nx
G = nx.Graph()
begin_sent = 22
end_sent = 24
sents = thursday_sents[begin_sent:end_sent+1]
for sent in sents:
G.add_edges_from(nltk.bigrams(w for w in sent if w.isalpha()))
nx.draw(G)
Lexical clusters in Dickens¶
Below [BKL] refers to "Natural Language Processing with Python" by Bird, Klein and Loper and [MAR] refers to "Mining the Social Web" by Matthew A. Russell.
Getting the raw text¶
See page 95 of [BKL].
In [129]:
import codecs, nltk, pprint
hard_times_path = "/home/matthew/workspace/resources/C/Corpus Stylistics/Dickens, Charles/786-0.txt"
david_copperfield_path = "/home/matthew/workspace/resources/C/Corpus Stylistics/Dickens, Charles/pg766.txt"
f = codecs.open(hard_times_path, encoding = 'utf-8')
david_copperfield_file = codecs.open(david_copperfield_path, encoding = 'utf-8')
In [27]:
hard_times_raw_text = f.read()
len(hard_times_raw_text)
Out[27]:
In [130]:
david_copperfield_raw_text = david_copperfield_file.read()
len(david_copperfield_raw_text)
Out[130]:
Sentence tokenizing¶
See page 112 of [BKL].
In [28]:
sent_tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')
sents = sent_tokenizer.tokenize(hard_times_raw_text)
In [29]:
print sents[171]
In [30]:
len(sents)
Out[30]:
An alternative approach based on [MAR].
In [31]:
sents = nltk.tokenize.sent_tokenize(hard_times_raw_text)
In [32]:
print sents[171]
In [131]:
DC_sents = nltk.tokenize.sent_tokenize(david_copperfield_raw_text)
print DC_sents[171]
Word tokenizing¶
In [33]:
tokens = [nltk.tokenize.word_tokenize(s) for s in sents]
In [34]:
len(tokens)
Out[34]:
In [35]:
print tokens[171]
In [134]:
DC_tokens = [nltk.tokenize.word_tokenize(s) for s in DC_sents]
In [135]:
print DC_tokens[171]
POS tagging¶
Warning: Very slow
In [36]:
# pos_tagged_tokens = [nltk.pos_tag(t) for t in tokens]
n-grams¶
In [63]:
' ' .join(tokens[171])
Out[63]:
In [61]:
bigram = nltk.bigrams(w for w in tokens[171] if w.isalpha())
In [62]:
import networkx as nx
G = nx.Graph()
G.add_edges_from(bigram)
nx.draw(G)
In [75]:
print nltk.ngrams((w for w in tokens[171] if w.isalpha()), 5)
Now consider the full text of "Hard Times", here represented by the list tokens.
In [67]:
fivegrams = []
for t in tokens:
fivegrams += nltk.ngrams((w for w in t if w.isalpha()), 5)
In [73]:
print ' '.join(fivegrams[2000])
Create a dictionary to count occurrences of specific 5-grams.
In [76]:
D = {}
for gram in fivegrams:
if not D.get(gram):
D[gram] = 1
else:
D[gram] += 1
Iterate through the keys of the dictionary and print out and 5-grams that have more than 3 occurrences in the text.
In [86]:
for gram in D.keys():
if D[gram] > 3:
print ' '.join(gram), D[gram]
Now we can try to find the occurrences of one of the 5-grams, 'vension with a gold spoon', in the original text.
In [87]:
hard_times_raw_text.find('venison with a gold spoon')
Out[87]:
In [116]:
print hard_times_raw_text[250010:250082]
In [96]:
hard_times_raw_text.find('venison with a gold spoon',250057)
Out[96]:
In [113]:
print hard_times_raw_text[250084:250182]
We only find 2, instead of the 4 we counted, though. Probably an issue of punctuation.
In [115]:
hard_times_raw_text.find('venison with a gold spoon', 250157)
Out[115]:
In [122]:
five_gram_2 = 'his hands in his pockets'
hard_times_raw_text.find(five_gram_2)
Out[122]:
In [123]:
hard_times_raw_text.find(five_gram_2, 39378)
Out[123]:
In [124]:
hard_times_raw_text.find(five_gram_2, 41789)
Out[124]:
In [125]:
hard_times_raw_text.find(five_gram_2, 58760)
Out[125]:
In [126]:
hard_times_raw_text.find(five_gram_2, 476236)
Out[126]:
In [127]:
hard_times_raw_text.find(five_gram_2, 514355)
Out[127]:
Now we will try to reproduce Table 3.3 on p.47 of [MM].
In [136]:
DC_bigrams = []
for t in DC_tokens:
DC_bigrams += nltk.ngrams((w for w in t if w.isalpha()), 2)
In [138]:
DC_D = {}
for gram in DC_bigrams:
if not DC_D.get(gram):
DC_D[gram] = 1
else:
DC_D[gram] += 1
In [139]:
for gram in DC_D.keys():
if DC_D[gram] > 500:
print ' '.join(gram), DC_D[gram]
In [140]:
DC_trigrams = []
for t in DC_tokens:
DC_trigrams += nltk.ngrams((w for w in t if w.isalpha()), 3)
In [141]:
DC_D_3 = {}
for gram in DC_trigrams:
if not DC_D_3.get(gram):
DC_D_3[gram] = 1
else:
DC_D_3[gram] += 1
In [142]:
for gram in DC_D_3.keys():
if DC_D_3[gram] > 60:
print ' '.join(gram), DC_D_3[gram]
In [144]:
DC_fourgrams = []
for t in DC_tokens:
DC_fourgrams += nltk.ngrams((w for w in t if w.isalpha()), 4)
In [145]:
DC_D_4 = {}
for gram in DC_fourgrams:
if not DC_D_4.get(gram):
DC_D_4[gram] = 1
else:
DC_D_4[gram] += 1
In [149]:
for gram in DC_D_4.keys():
if DC_D_4[gram] > 18:
print ' '.join(gram), DC_D_4[gram]
By comparison with [MM] there is a problem in how I am handling punctuation. For example, I record the most frequent four gram as "I do know what" but in [MM] it is "I don't know what".
In [163]:
david_copperfield_raw_text.find("I do know what")
Out[163]:
In [151]:
david_copperfield_raw_text.find("I don't know what")
Out[151]:
In [162]:
print david_copperfield_raw_text[19264:19296]
At the very least I ought to reduce all text to lowercase, as done in [MM].
Finding suspended quotations¶
We begin with an extract.
In [40]:
test_extract = sents[1024: 1037]
In [41]:
print ' '.join(test_extract)
Interview Task¶
In [2]:
import codecs, nltk
little_dorrit_path = "/home/matthew/workspace/resources/C/Corpus Stylistics/Dickens, Charles/pg963.txt"
f = codecs.open(little_dorrit_path, encoding = 'utf-8')
little_dorrit_file = codecs.open(little_dorrit_path, encoding = 'utf-8')
In [3]:
little_dorrit_raw = little_dorrit_file.read()
In [4]:
len(little_dorrit_raw)
Out[4]:
In [5]:
little_dorrit_raw.find(u'At the close of this recital')
Out[5]:
In [9]:
end_phrase = 'producing the money.'
little_dorrit_raw.find(end_phrase)
Out[9]:
In [106]:
task_string = little_dorrit_raw[1725461:1727185 + len(end_phrase)]
print task_string
In [397]:
import re
re_1 = r"'[^']+'"
re_2 = r"'[a-zA-Z0-9_,!? ]+(?:[-'][a-zA-Z0-9_,!? ]+)*'"
re_3 = r"'[^']+[\.,!?]'"
nltk.re_show(re_1, task_string[423:])
In [396]:
re.findall(re_1, task_string)
Out[396]:
Using the NLTK built-in corpus reader (optional)¶
In [290]:
from nltk.corpus import PlaintextCorpusReader
corpus_root = "/home/matthew/workspace/resources/C/Corpus Stylistics/Dickens, Charles"
wordlists = PlaintextCorpusReader(corpus_root, '.*')
wordlists.fileids()
Out[290]:
In [42]:
In [43]:
len(hard_times_sents_raw)
In [ ]:
a = 0
for sentence in hard_times_sents_raw:
a += len(sentence)
print a
In [ ]:
import re
s = hard_times_sents_raw[1000]
# re.findall(r'\W+', sentence)
print s
Removing extraneous text (optional)¶
In this section I show how to remove extraneous text from the raw string for "Hard Times". This is done by hand by first identifying the first and last sentences of the text. There are some issue about Unicode that I haven't yet resolved.
In [ ]:
hard_times_first_sentence = '\xe2 \x80\x98 NOW , what I want is , Facts .'
hard_times_first_sentence.split() in hard_times_sents_raw
In [ ]:
first_sentence_index = hard_times_sents_raw.index(hard_times_first_sentence.split())
In [ ]:
' '.join(hard_times_sents_raw[first_sentence_index])
In [ ]:
hard_times_last_sentence = 'We shall sit with lighter bosoms on the hearth , to see the ashes of our fires turn gray and cold .'
hard_times_last_sentence.split() in hard_times_sents_raw
In [ ]:
last_sentence_index = hard_times_sents_raw.index(hard_times_last_sentence.split())
In [ ]:
' '.join(hard_times_sents_raw[last_sentence_index])
In [ ]:
hard_times_sents = hard_times_sents_raw[first_sentence_index:last_sentence_index + 1]
In [ ]: