Lecture 11: Natural Language Processing (NLP) [FINISHED]

In this lecture we're going to shift gears from dealing with numerical data to text data.

Working with text as data is known as Natural Language Processing (NLP). A common use of NLP is categorizing a set of text. Perhaps the most ubiquitous example is a spam filter. It reads the text of a message and determines if it is "spam" or "ham."

We'll employ one simple NLP algorithm known as the bag-of-words algorithm to classify SMS messages as spam. There are more sophisticated methods (see end of the lecture) but this will give us the big idea. More-sophisticated methods typically amount to tweaks on how we process the data or more complex classifier models.

We will use the natural language tool kit (nltk) to help us process the text data. It comes with annaconda, but if you need to install it:

pip install nltk

We'll also be using word clouds to visualize identification so install this as well:

pip install wordcloud

The agenda for today's lecture is as follows:

1. Motivation: Spam messages

2. The bag-of-words model

3. Developing a model to identify spam messages

4. Term Frequency Inverse Document Frequency (TF-IDF)

1. Motivation: Spam Messages ¶

Spam emails/messages belong to the broad category of unsolicited messages received by a user. Spam occupies unwanted space and bandwidth, amplifies the threat of viruses, and in general exploits a user’s connection to social networks. Plus, they're annoying.

Our goal is to classify a message as spam (unwanted message) or ham (wanted message).

Languages are harder for algorithms to interpret and analyze than numeric data since:

1. Sentences are not of fixed lengths, but most algorithms require a standard input vector size.

2. Most algorithms cannot understand words as input: hence, each word needs to be represented by some numeric value.

So our method is:

1. Pre-processing: Clean up the text. This is the new stuff.

2. Estimate a classifier model on the training data: Let $\text{word}_{ji}$ be the number of times the word $j$ occurs in message $i$

3. Test the model on the testing data.

4. Use the estimated model to filter incoming messages

2. The bag-of-words model ¶

A bag-of-words model allows us to extract features from textual data. Since an algorithm doesn't understand language, we need to use a numeric representation for the words. This numeric representation can later be fed to any algorithm for further analysis. There are many ways to do this, bag-of-words is the simplest and provides a good foundation for working with text.

The model is called "bag-of-words" because the order of the words or the structure of the sentence is lost in this model. Only the occurrence or presence of a word matters. Hence, we can think of the model in such a way:

• we have a big empty bag
• we have a vocabulary (ie, a text or corpus).

We pick up words one by one and put them in the bag, adding to the frequency of their occurrence. We then select the most common words as features for passing through our algorithm of choice. We can therefore view our approach as identifying documents which share similar kinds of words.

Here is an example:

Even though python is a good with text, we will still need to convert our text into some numeric data to get a classifier model to analyze it. Let's create a matrix with the word counts. Each row of the matrix is an observation (a message) and each column is a word. The cells in the matrix are the number of times that word is found in the message.

scikit gives us the CountVectorizer to do this for us.

Note that X is an array-like object. We are in the realm of scikit, which doesn't use DataFrames. Let's turn this back into a DataFrame, though, so we can see things clearly.

The X array contains our features and the data['Label'] column contains our outcome variable. We now have the data ready to estimate a classifier model (e.g., a logit regression).

Note that this methdology of turning text into data is not limited to classification problems. For example, we could use this approach to connect stock performance with FOMC statements to predict how the Federal Reserve's statements on the economy influence the S&P500, the Dow Jones, individual stocks, and government treasury prices. NLP is a broad topic and a lot of fun.

This dataset is too small to actually fit a model, so let's move on to something bigger.

3. Developing a model to identify spam messages ¶

The dataset that we are using is an SMS spam collection dataset. It contains over 5,500 messages in English. There are two columns. The first column corresponds to the actual text message. The second column tells us whether the text is 'ham' or 'spam'.

Data Pre-Processing¶

This is the part that makes nlp different from working with numeric data. We need to clean up the text and turn it into a feature matrix.

'I am helping raise $100 for UGA Athens'                     original
'i am helping raise $100 for uga athens'                     homogenize spelling
'i am helping raise for uga athens'                          remove non-alphabetic characters
['i', 'am', 'helping', 'raise',  'for', 'uga', 'athens']     tokenize
['helping', 'raise', 'uga', 'athens']                        remove stop words
['help', 'raise', 'uga', 'athens']                           stem and lem
'help raise uga athens'                                      back to a single string

Then create the feature matrix

1. Homogenize the capitalization¶

We don't want to worry about 'Hello' not being equal to 'hello'. Let's make everything lowercase.

2. Remove non-alphabetic characters¶

Our algorithm will only use words to characterize messages. This is not necessary, but it simplifies our approach today. Perhaps messages with numbers in them are more likely to be spam?

We will remove them using a regular expression. We have not covered 'regex' (there is never enough time!) but regex is a powerful string search language that is a part of python and most other programming languages. I have a notebook on regex here which you can work though if you are interested.

The code to remove the non-alphabetic characters is

dataset['message'].str.replace('[^A-Za-z]', ' ', regex=True)

The regex part is the '[^A-Za-z]'. It says: "find everything that is not the letters A through Z or a through z." We replace the non-alphabetic stuff with a space.

3. Tokenize the strings¶

Break the stings up into lists of words, which are easier to process. This is very similar to using .str.split(' '). Here we use the tokenizer method from nltk. It is a bit more sophisticated than a simple split.

from nltk.tokenize import word_tokenize as wt

We also need to download the punctuation dataset.

4. Removing stop words¶

Now we eliminate stop words—words in the text that add no specific meaning. They often involve prepositions, helping verbs, and articles (e.g., in, the, an, is). Since these add no value to our model, let's get rid of them.

Fortunately, linguists have already identified stopwords so we can readily identify and exclude them

from nltk.corpus import stopwords
stop_wrds = stopwords.words('english')

stop_wrds is a list of English-language stop words.

We need to loop through the lists and check for stop words. I will write a small function that does the looping and then apply it to the DataFrame's column using .apply().

Again, we need to download the stopwords, first.

5. Stemming and Lemmatization¶

Words like act, actor, and acting are all versions of the same root word (act). Stemming and lemmatization are techniques used to truncate words in order to get the stem or the base word. The difference between these two methods is that after stemming, the stem may not be an actual word, whereas lemmatization always produces a real world, which results in better interpretation of the corpora by humans.

For example, studies could be stemmed as studi (not a word), but will be lemmatized as study (an existing word). To be honest, this feels like a rabbit hole so I'm treating this stuff as a block box and trusting that the linguists are doing a good job.

Let's stem these words.

That seemed like a lot of work, but it always does when we are first learning something. Putting all the code together, the processing is simply:

dataset['message'] = dataset['message'].str.lower()
dataset['message'] = dataset['message'].str.replace('[^A-Za-z]', ' ', regex=True)
dataset['message'] = dataset['message'].apply(wt)
dataset['message'] = dataset['message'].apply(remove_stops)
dataset['message'] = dataset['message'].apply(stem_it)

You could even wrap all that up in a function, too...

Create the feature matrix¶

We are done pre-processing.

1. Turn the lists of words back into strings.
2. Create the feature matrix using CountVectorizer.

Visualizing Keywords¶

We've seen that data visualization is a handy way of better understanding data variation and that's still true here though the visualization tools will be different. For example, we can make a wordcloud, which represents most common words in a space, with the size of each word proportional to the frequency of its occurrence. To do so, we need to add a new package:

pip install wordcloud

Estimate the logit model¶

Nothing new here.

1. Split data into test/train
2. Estimate on training data
3. Test on, well, testing data

Not too shabby!

Practice¶

We're going to practice using the "20 Newsgroup" data set which is

a collection of approximately 20,000 newsgroup documents, partitioned (nearly) evenly across 20 different newsgroups. To the best of my knowledge, it was originally collected by Ken Lang, probably for his "Newsweeder: Learning to filter netnews" paper, though he does not explicitly mention this collection. The 20 newsgroups collection has become a popular data set for experiments in text applications of machine learning techniques, such as text classification and text clustering.

This data set is located within scikit learn but I've created a csv file you can load.

1. Download, unzip, and load the file 'newsgroups.csv'. Only import the first 500 rows, Try the nrows option of .read_csv().

   article is the message. category code is the newsgroup category code. category is the newsgroup category name.

   Our goal: Create a classifier that predicts the category code of an article.

2. Make sure "article" is a string. Use .astype(str).

3. How many articles are there in each category. Looks like it's time for .groupby().

4. Process the text data. All the code to do this is gathered in the cell above the Create feature matrix heading.

4. Turn the lists of words in article into strings using .str.join(' ')

5. Create the feature matrix.

6. Create the outcome variable. (the Series that contains the category codes)

7. Create your testing and training datasets.

8. Estimate a logit model.

9. Check the accuracy of your model by omputing the .score().

10. Go back to step 1. and increase the number of rows to 1000. Rerun your code. Does the accuracy improve?

11. Try a Random Forest model. Set the n_estimators=100 and random_state=0.

12. There are about 20,000 rows in the data file. You can keep adding data, but things are going to get slow. This is why big data projects need special techniques and big computers. That being said, the "big" ideas are the same -- we're just operating at a bigger scale.

4. Term Frequency Inverse Document Frequency (TF-IDF) ¶

Let's try adding a little more intelligence to the words count methodology by replacing bag-of-words with TF-IDF (Term Frequency Inverse Document Frequency) to account for the importance of words. This is very common algorithm to transform text into a meaningful representation of numbers which is used to fit machine algorithm for prediction. The statistic is designed to better define how important a word is for a document, while also taking into account the relation to other documents from the same corpus. This is performed by looking at how many times a word appears into a document while also paying attention to how many times the same word appears in other documents in the corpus.

Let's dig into the details a bit.

TF-IDF consists of two parts :

1. Term Frequency (TF): how many times a word appears in a document
2. Inverse Document Frequency (IDF): Inverse document frequency of the word across a set of documents.

Let's demonstrate the idea with the example from before:

To find "term frequency" count the number of times each unique word occurs in the text. This is where we stopped with Bag-of-Words.

Term frequency tends to give more weightage to common words (eg, "is", "if", "the") so we need something to reduce this effect since common words won't be useful to identify a pattern in the document. For example, if we had a document on sports and another on medicine, words like "Football" and "Hypertension" will be rare and separate (identify) the documents. Zipf's Law tells us that the frequency of any word is inversely proportional to its rank in a frequency table. Thus the most frequent word will occur approximately twice as often as the second most frequent word, three times as often as the third most frequent word, etc.

Accordingly, IDF applies more weight to words which occur rarely.

The TF-IDF score is then simply the interaction of the two.

Scikit-learn then normalizes these values using the l2 norm:

That's it. Now use these features to understand some outcome variable of interest in a ML model.

Important: We did TF-IDF by hand in the code above but you can incorporate TF-IDF easilly by importing TfidfVectorizer via

from sklearn.feature_extraction.text import TfidfVectorizer

Then use TfidfVectorizer() instead of CountVectorizer in your code.

Practice¶

1. Develop the newsgroup model again using the TF-IDF algoorithm and logistic model for the first 500 rows. Does this technique improve the model's effectiveness at classifying the articles?