Rosetta¶
Tools for data science with a focus on text processing.¶
- Focuses on "medium data", i.e. data too big to fit into memory but too small to necessitate the use of a cluster.
- Integrates with existing scientific Python stack as well as select outside tools.
Tools and utilities¶
cmd¶
- Unix-like command line utilities. Filters (read from stdin/write to stdout) for files
parallel¶
- Wrappers for Python multiprocessing that add ease of use
- Memory-friendly multiprocessing
text¶
- Stream text from disk to formats used in common ML processes
- Write processed text to sparse formats
- Helpers for ML tools (e.g. Vowpal Wabbit, Gensim, etc...)
- Other general utilities
workflow¶
- High-level wrappers that have helped with our workflow and provide additional examples of code use
modeling¶
- General ML modeling utilities
Lets begin¶
In [1]:
import os
import pandas as pd
Text Processors¶
Streamers¶
- Data in many text processing problems comes in the form of
- flat files
- repeated calls to an DB or API
- some 'online' stream
- A lot of these can be handled streaming the data either from disk, DB, API minimizing CPU use
- In addition, a lot of streaming is embarassingly parallel so can be easily scaled
In [2]:
#all you realy need to know is that CABLES is the directory where the data (or cables)
#are stored on your machine
DATA = os.environ['DATA']
CABLES = os.path.join(DATA, 'declass', 'cables_short')
RAW = os.path.join(CABLES, 'raw')
PROCESSED = os.path.join(CABLES, 'processed')
SPARSE = os.path.join(CABLES, 'sparse')
sfile_path = os.path.join(SPARSE, 'cables-short.vw')
filtered_sfile_path = os.path.join(PROCESSED, 'cables-short-filtered.vw')
sff_path = os.path.join(PROCESSED, 'sff.pkl')
Streaming: given a collection of objects streaming is the paradigm of processing these objects one at a time in memory, extracting relevant information, writing the information, and discarding the original object¶
Note: after a streaming process is complete, the original collection should no longer be needed for the analysis at hand¶
- Lets write a simple file streamer
In [16]:
#filefilter is a module which helps with basic file/dir functions, such as
#retrieving all paths from a given directory and it's subdir's
from rosetta.text import filefilter
In [17]:
def simple_file_streamer(base_path):
paths = filefilter.get_paths(base_path, get_iter=True)
for p in paths:
with open(p) as f:
text = f.read()
yield(text)
In case you haven't worked much with iterators explicitely, here is a small refresher...¶
- For those familiar with generator functions, or iterators, you'll notice that this is exactly what we mean by "streamer," i.e. anything that retrieves files or extracts information from therein and has a .next() method
- python docs have a short intro about generators (http://docs.python.org/2/tutorial/classes.html#generators)
In [5]:
def my_iter(N):
i=0
while True:
if i == N:
raise StopIteration
else:
yield i
i += 1
In [6]:
mi = my_iter(5)
In [7]:
mi.next()
Out[7]:
In [8]:
#note the raised StopIteration; lets see how a for look handles this
for i in my_iter(5):
print i
In [18]:
simple_stream = simple_file_streamer(RAW)
In [19]:
#lets look at what this object is
type(simple_stream)
Out[19]:
Note: this is the first time anything is read into memory¶
In [20]:
#lets see what the .next() yields (and splitlines to make it more readable)
simple_stream.next().splitlines()
- Now since our end goal is to build a topic model we probably want to have a more fexible streamer, i.e. one that can return file ids, text or tokens (based on some predefined tokenizer)
- luckily we have one such streamer written
In [4]:
from rosetta import TextFileStreamer, TokenizerBasic
text_streamer = TextFileStreamer(text_base_path=RAW, file_type='*',
tokenizer=TokenizerBasic())
In [3]:
from rosetta.text import streamers
In [5]:
stream = text_streamer.info_stream()
In [8]:
stream.next()
Out[8]:
In [15]:
#lets take a quick look at TextFileStreamer
TextFileStreamer?
- Note: you can pass a tokenizer function to TextFileStreamer(), i.e. any function that takes a string of text and returns a list of strings (the "tokens")
- We have written a basic tokenizer function and class to add functionality and because the nltk.word_tokenize() was slow
- It also has a few other nice options such as shuffle, file_type, etc and a bunch of methods
In [9]:
text = stream.next()['text']
In [10]:
print text
text_streamer.tokenizer.text_to_token_list(text)
text_streamer.tokenizer.text_to_counter(text)¶
In [11]:
#lets look at a few methods
token_stream = text_streamer.token_stream() # returns a generator function which yields a stream of tokens
In [16]:
token_stream.next() # this is what our basic tokenizer returns (we are skipping stop words and numerics by default)
Out[16]:
text_streamer.doc_id # returns a list of retrieved doc ids etc
In [50]:
#if you want to use another tokenizer it's easy
import nltk
nltk.word_tokenize(text)
text_streamer_nltk = TextFileStreamer(text_base_path=RAW, file_type='*',
tokenizer_func=nltk.word_tokenize)
In [51]:
stream_nltk = text_streamer_nltk.token_stream()
In [59]:
stream_nltk.next()
Out[59]:
Vowpal Wabbit for LDA Topic Modeling¶
- LDA = Latent Dirichlet Allocation
- treats each document as a bag of words
- a topic is chosen from a topic distribution $p(k)$ where $k=1, \dots , K$
- a word is chosen from the k'th topic distribution $p(w|k)$ and thrown into the bag
- distrubutions $p(k)$ depends on $\theta$ ~ $Dir(\alpha)$ and $p(w|k)$ depends on $\beta$ a $k\times V$ matrix of word probabilties
- these 'latent' variables are chosen to maximize the probability of producing the observed documents, and in turn depend on user chosen parameters $\alpha$ and $\eta$
- the model produces two important probability distributions:
- $p(w|k)$, the probability of $w$ bring generated by topic $k$ and
- $p(k|d)$, the probabilty of topic $k$ being used to generate a randomly chosen word from document $d$
- these topic and word weights can be used to understand the semantic structure of the documents as well as generate document feature
- for more details about LDA topic modeling see the wonderful Blei, Ng, and Jordan paper
- Vowpal Wabbit
- optimized (very fast) C++ library http://hunch.net/~vw/
- can find tutorials at https://github.com/JohnLangford/vowpal_wabbit/wiki/Tutorial and https://github.com/columbia-applied-data-science/rosetta/blob/master/examples/vw_helpers.md
In [17]:
from rosetta.text import text_processors, filefilter, streamers, vw_helpers
In [18]:
#create the VW format file
my_tokenizer = text_processors.TokenizerBasic()
stream = streamers.TextFileStreamer(text_base_path=RAW, tokenizer=my_tokenizer)
stream.to_vw(sfile_path, n_jobs=-1, raise_on_bad_id=False)
In [ ]:
### somewhere here run (stick with 5 passes or so...)
# rm -f *cache
#vw --lda 20 --cache_file doc_tokens.cache --passes 5 -p prediction.dat --readable_model topics.dat --bit_precision 16 --lda_D 975 --lda_rho 0.1 --lda_alpha 1 ../sparse/cables-short.vw
In [19]:
#load the sparse file
formatter = text_processors.VWFormatter()
sff = text_processors.SFileFilter(formatter)
sff.load_sfile(sfile_path)
In [63]:
#remove "gaps" in the sequence of numbers (ids)
sff.compactify()
sff.save(PROCESSED + '/sff_basic.pkl')
In [66]:
sff.to_frame().sort_index(by='doc_fraction', ascending=False).head(10)
Out[66]:
Look at LDA results¶
In [20]:
#use the LDAResults class from rosetta to convert back to readable, python friendly formats
lda = vw_helpers.LDAResults(PROCESSED + '/topics.dat',
PROCESSED + '/prediction.dat', PROCESSED + '/sff_basic.pkl')
In [21]:
#look at some of the words
topic_words = lda.pr_token_g_topic.loc[:,'topic_12'].order(ascending=False).index[:10]
lda.sfile_frame.loc[topic_words]
Out[21]:
In [22]:
#look at the topics themselves
lda.print_topics(10)
In [23]:
##
lda.pr_topic_g_doc.T.loc[[0]].plot(kind='bar', figsize=(20,10),
title = 'First Document Topic Weights')
Out[23]:
In [30]:
#or at the average topic probabilties
import random
r = lambda: random.randint(0,255)
my_colors = ['#%02X%02X%02X' % (r(),r(),r()) for i in range(20)]
#my_colors = 'rgbkymc'
lda.pr_topic_g_doc.mean(axis=1).plot(kind='bar', figsize=(15,10), color=my_colors,
title='Average Topic Probabilities')
Out[30]:
Have fun!¶
In [ ]: