Usupervised classification of philosophical genres¶

This notebook is a part of work being done for the Trace of Theory project, a collaboration between researchers of NovelTM and the HathiTrust Research Center (HTRC).

Here, we'll use unsupervised techniques to identify clusters of similar texts within a corpus of about 3,200 philosophical texts. These texts were previously identified in the HathiTrust public domain corpus using a list of philosophical keywords. The idea now is to look for something like philsophical "genres" within this subcorpus of philosophical texts and to compare the computational results to human labels. Our features will mix word-count data with measures of form and with textual metadata, so that we're examining not just subject matter, but also style and (minimal) context.

The work below is almost exclusively about methods. There's not a lot of analysis, and the notebook ends with suggestions for things to try, rather than conclusions about philosophical genre.

Roadmap¶

• Download feature data for the 3,200 philosophical texts from the HathiTrust Research Center
• Parse feature data
• Calculate other features derived from the same sources
• Reduce the dimensions of the feature space in order to have some hope of clustering the texts
• Perform k-means and DBSCAN clustering on the dimension-reduced features
• Visualize the clustering output alongside the human labels; use both static plots (via Matplotlib/Seaborn) and interactives (via Bokeh)

Imports¶

Get input files¶

In previous work, Geoffrey Rockwell and others used a list of philosophical keywords to identify texts from the HTRC corpus that were likely to be philosophical. We use the resulting list of 3,231 philosophical texts. From those we just want basic metadata and HTIDs in order to download the corresponding HTRC derived feature files. Note that we're using a 'clean,' UTF-8 encoded version of the list of texts; the original list causes problems down the line.

In the data, 'freq' is the relative frequency of philosophical keywords. The field for 'disc' (here renamed to 'label') is a human-applied label for the type of philosophy in each volume. The labels are distributed like so:

OK, so really only about 25% of the 'philosophical' texts are labeled philosophy proper. We'll bear this in mind. Now we want to download the HTRC derived feature files for those 3,231 volumes. To do that, we'll generate a list with full pairtree file paths, then use rsync to bring them over to local storage. See the HTRC documentation for details concerning their storage structure.

Note that HTIDs need to be convernted to filesystem-safe versions by converting instances of ':' to '+' and '/' to '+'.

Now parse the file-safe HTIDs into pairtree paths (as used by the HTRC). In a different world, I suppose we'd use a pairtree library rather than handling this from scratch ...

Call rsync to retrieve the feature files from the HTRC. Note that a few files (32 out of 3,231) will fail. One could examine these by printing the failures list.

If you examine the list of failures, you can see that some of the feature files are missing. But not many. We'll deal with this later by excluding the missing files.

Extract features from texts¶

We'd like to use Peter Organisciak's htrc-feature-reader library to extract data from the feature files, but it isn't working at the moment. Switch to that library in the future, since it's easier than the hand-coding below.

In the end, we want to extract and/or compute the following features for each volume:

• TF-IDF scores for the top, say, 5,000 words. Then reduce dimensionality for these features via PCA
• Length of volume in words
• Date of publication
• Reading level score, which requires knowing average sentence length in words and average syllables per word
• Diction score, as measured by proportion of tokens that entered the English language before 1150. Higher values here correspond to "lower" diction (i.e., more Germanic, less Latinate).
• Fraction of tokens that are proper nouns (how character-driven is the text?)
• Fraction of tokens that are verbs (how action-oriented is the text?)

The idea isn't that these are the only -- or even necessarily the correct -- features to use. But they combine content analysis with formal and metadata features is a way that may be illustrative.

First, make a pass over all the files, eliminating from the data frame any that did not download from HTRC. This isn't super efficient, but it's easier than keeping track while we're also calculating features in the next step.

Note that we've gone from 3231 entries to 3199; there were 32 feature files that failed to download from HTRC. That's good enough for now.

Define a function that calculates the desired volume-level features from an HTRC compressed feature file.

Extract derived feature data from HTRC files¶

This is slow; it takes about half an hour on my laptop.

OK, we capture 27.2% of the observed variance in 5,000 tfidf terms with these 10 components. Is this good? Well, on one hand, we lose almost 3/4 of the underlying word-frequency information. On the other hand, we've reduced our dimensionality by 99.8% (from 5,000 dimensions to 10) and still retained more than a quarter of the info, so ...

Some quick vis for reference¶

Plot histograms of corpus data.

Hmm ... that's a bit worrying. What's with all the short books? Take a closer look ...

Meh, guess there are just a lot of pamphlets in the dataset. Moving on.

Reading levels more or less normally distributed. These are not easy-reading books.

Relatively normal distribution of diction scores.

Ditto verb fractions.

Not at all normal distribution of proper nouns; long tail into high values.

Books skew heavily toward recent publication. All data is drawn from the HathiTrust public domain corpus, so there's nothing after 1923, hence the wall at 1920.

Looks like there was an explicit cutoff in the keyword-based selection process at philosophical word frequency = 0.10. Result is the clipped high side of what I presume is an underlying normal distribution of these terms in the full corpus. OK, no problem.

OK, all our data is in place, though it's not (yet) normalized. Time to do some clustering and visualization.

Clustering¶

We'll try both k-means and DBSCAN ...

Note that the normalized mean is zero and standard deviation is 1. There are 3165 documents, each having 17 features (10 word-based PCs and 7 form/metadata features).

Note, FWIW, that k-means clustering is very fast. DBSCAN is slower, but not radically so. The slow steps in all this work, as you'll already have seen if you've run the code yourself, involve acquiring and wrangling the data, not working with it once it's nicely ingested.

Cluster -1 in DBSCAN output means 'Other', i.e., noise. This suggests one of two things: either we don't have the right value of epsilon or our textual data is insufficiently dense in the feature space.

Back of the envelope calculation re: density ... We're working in 17 dimensions, and we have 3,165 texts. Assume each dimension can take only binary values, which is to say that it divides the possible clustering space in half. That means we have $2^{17}$ = 131,072 logical bins into which to place the texts. So, ummm, no wonder we don't really see good clustering over just 3,165 texts. The chance of seeing even two or three texts in the same bin is pretty small.

This is an obvious area for improvement. What would be an appropriate number of dimensions in whch to cluster c. 3,000 points? Well, we want more bins than clusters, so that we don't just end up with unrelated things being stuffed into the same bin for lack of alternatives. But we don't want so many bins that none of them has any significant number of texts. Maybe, say, 300-ish bins, for an average of 10 items per bin? $2^8$ = 256, which is in the right ballpark.

The thing to do, then, would be to use PCA (or another dimensional reduction technique, like MDS or t-SNE) to bring our dimensions down to 8. Not doing this here, but the idea is exactly the same as what we did to reduce the 5,000 TF-IDF features to 10, or how we reduce the current 17 dimensions to 2 for plotting below.

Plots¶

Human labels¶

Recall the distribution of human labels in our data. Remember that 'x' and '?' are junk or miscellaneous categories; we don't expect them to correspond to known-good philosophical subgenres.

k-means¶

NB. Labels aren't correlated with human labels, so this IS NOT a plot of performance vs. humans. Don't read anything into the colors or ording of clusters.

DBSCAN¶

Ditto re: lack of correlation here. Note that cluster -1 means "no cluster" or "outlier."

Plot in PCA space to see the clusters¶

Human labels¶

This mostly shows that there's little detectable differentiation between the various subgenres of philosophy in 17-dimensional feature space.

Human labels in Bokeh with interactivity¶

Here's the same thing, but plotted using Bokeh to give us interactivity. Note, too, that we're now leaving out 'x' and '?' types, for a bit of additional clarity.

The first plot here uses Bokeh's high-level plotting interface, which is really easy, but doesn't allow for the complex mouse-over information that we'll add in the second plot.

If there's no inteactive visualization above, it's probably because you're viewing this notebook in a viewer rather than running the code live. Sorry!

Now the same info, but using Bokeh's "mid-level" plotting API to add more complex mouse-over behavior, better color and alpha support, etc.

Again, there should be a cool interactive vis above, including points labeled on mouseover, colored by class, and arbitrarily zoomable. If you don't see it, you'll need to download and run this notebook on your own machine.

k-Means labels¶

DBSCAN labels¶

To do¶

Things that one might try to improve these results:

• Perform grid search on DBSCAN to find best parameters. Currently non-optimal; see few large clusters and tendency to mark many as noise.
• Dimensionality is very likely too high overall. Reduce feature space prior to clustering.
• Correlate human labels with machine labels. Maybe just by majority vote vs. human labels.
• Part of any future analytical work will involve examining the PCA loadings to see what's driving texts in certain directions.
• Extend the same analsysis to literary criticism, which is also part of the "Trace of Theory" project mandate.