Lecture 4: Classification [FINISHED]

Classification is a task that requires the use of machine learning algorithms to learn how to assign a class label to examples from the problem domain. We may want to classify:

• email as "spam" or "not spam".
• a tumor as "benign" or "malignant".
• a flower as a particular species.
• which car a person is likely to buy.

The first two example are binary -- the outcome is either true or not. The second two examples are multi-class -- there are several different discrete values the outcome could take.

There are many different types of classification tasks that you may encounter in machine learning and specialized approaches to modeling that may be used for each. In this lecture we'll touch on some of the more popular:

1. Binary Classification Models
   1.1 Load and Explore Data
   1.2 Logistic
   

1. Multi-Class Classification Models
   2.1 Load and Explore Data
   2.2 Logistic
   2.3 Naive Bayes
   

Process
While each model will have nuances, the process for fitting all of these models will be similar.

(i) Initialize the model by creating a model object.
(ii) Fit the model on training data.
(iii) Test model accuracy by comparing test data predictions with test data results.

Note: Each model has what are called "hyperparameters" which we could "tune" using cross-validation as in the last lecture. For expediency, we won't do that in this lecture but you should in a real-world application.

Class Announcements

None.

1. Binary Classification Models (top)¶

When the outcome variable is either true (1) or not (0), we can predict the outcome with a binary classification model. Let's load some data and get started.

1.1 Load and Explore Data on Breast Cancer (top)¶

sklearn comes with a bunch of pre-loaded datasets. You can see the available datasets here. Some of these we've already used in their csv form. Let's load the data on breast cancer by using the following function to transform the data into a dataframe. This isn't required to fit our models but let's stick with pandas since it's so powerful.

The data descriptions are here. Let's take a look at the data:

Split the Data into Training and Testing Data Sets¶

1.2 Logistic Regression (top)¶

The logistic regression passes the linear model through a non-linear function that constrains the output to lie between zero and one. In the logistic case, the function looks like

$$\text{Prob}(Y=1|X) = \frac{\exp \left({\beta_0+\beta_1 X}\right)}{1+\exp \left({\beta_0+\beta_1 X}\right)}.$$

The classifier gives us a set of outputs or classes based on probability when we pass the inputs through the above prediction function and returns a probability score between 0 and 1. We're currently considering the binary case where Y=1 or Y=0. We ecide with a threshold value above which we classify values into Class 1 and of the value goes below the threshold then we classify it in Class 2. Here, we predict the outcome variable is true whenever $\text{Prob}(Y=1|X) \ge 0.5$. Graphically, this amounts to:

When we fit the model to the training data, the algorithm chooses $\beta$ such that:

$$ \hat\beta = \max_\beta \sum_i \Big[y_i\log\big(f(x_i,\beta)\big) + (1-y_i)\log\big(1-f(x_i,\beta)\big)\Big] $$

where

$$ f(x_i,\beta)=\frac{\exp \left({\beta_0+\beta_1 x_i}\right)}{1+\exp \left({\beta_0+\beta_1 x_i}\right)} $$

What's happening here? In the first equation, the optimal $\beta$ generates predictions from $f(X,\beta)$ as consistent as possible with the observed ouctomes (y).

Some things to note:

• The objective function in the first equation is called a log-likelihood since it represents the model's ability the "likelihood" of predicting the outcome (y).
• The computer actually solves for the minimum so replace the $\max$ operator with the $\min$ operator and put a negative in front of the $\sum$ to convert the equation to what the computer is atually doing.
• In the math above there is only one feature in X but of course that's just a simplification. There can be many features!

Using Logistic Regression to Diagnose Cancer¶

Let's see how well the logistic model diagnoses breast cancer.

Wow, that worked really well! Let's try a different model and see if we can do better.

1.3 Testing Results¶

Confusion Matrix¶

We can evaluate how well the model is doing by decomposing the successes and failures using a tool called a "confusion matrix." A confusion matrix is a table which we use describe the performance of a classification model (or "classifier") on a set of test data for which the true values are known.

Let's look at the confusion matrix in our Logit ML:

Diagonal entries are when the model was successful. As for errors:

• 18 times the model predicted the tumor was malignant when it fact it was benign ("False Positive" or "Type I Error")

• 13 times the model predicted the tumor was bening when it fact it was malignant ("False Negative" or "Type II Error")

We can convert the confusion matrix into useful statistics:

• Precision is the ratio tp / (tp + fp) where tp is the number of true positives and fp the number of false positives. The precision is intuitively the ability of the classifier to not label a sample as positive if it is negative (i.e., avoid Type 1 errors).

• Recall is the ratio tp / (tp + fn) where tp is the number of true positives and fn the number of false negatives. The recall is intuitively the ability of the classifier to find all the positive samples.

• F-beta score can be interpreted as a weighted harmonic mean of the precision and recall, where an F-beta score reaches its best value at 1 and worst score at 0.

We can access all of these statistics in the classification_report.

The receiver operating characteristic (ROC) curve is another common tool used with binary classifiers. The dotted line represents the ROC curve of a purely random classifier; a good classifier stays as far away from that line as possible (toward the top-left corner). Area under the ROC Curve (AUC) is a way of translating the figure into a statistic where bigger equals better: A model whose predictions are 100% wrong has an AUC of 0 while a model whose predictions are 100% correct has an AUC of 1.

As you will see, this classifier does better than random guesses.

2. Multi-Class Classification Models (top)¶

Data may be discrete and also non-binary. In this section we discuss models to deal with multiclass discrete outcomes. Note also that any of these can be used in the binary case.

In a previous lecture we used the logit model to classify flower species based on the dimensions of its petal and sepal. Let's start there and then introduce additional tools.

2.1 Load and Explore Data on Flowers (top)¶

2.2 Logistic Regression Redux (top)¶

As we've seen before, the Logistic model can be used in multicase contexts as well. Let's fit the model again to the flower data.

Practice ¶

1. Split the data into training and testing data sets. Set the test_size=0.2 and the random_state=10.

1. Fit the logistic model to the training data.

1. Use the testing data to identify how well the model performs.

1. Describe the observations which are misclassified. Are there any trends?

We can generate ROC-type figure with multiclass as well by translating the multi-class into a binary problem. There are two ways to do this

OvR — One vs Rest OvR, or "One vs Rest", is a method to evaluate multiclass models by comparing each class against all the others at the same time. In this scenario we take one class and consider it as our "positive" class, while all the others (the rest) are considered as the "negative" class.

By doing this, we reduce the multiclass classification output into a binary classification one, and so it is possible to use all the known binary classification metrics to evaluate this scenario.

We must repeat this for each class present on the data, so for a 3-class dataset we get 3 different OvR scores. In the end, we can average them (simple or weighted average) to have a final OvR model score.

OvO — One vs One OvO stands for "One vs One" and is really similar to OvR, but instead of comparing each class with the rest, we compare all possible two-class combinations of the dataset.

2.3 Naive Bayes¶

Bayes' Theorem provides a method to calculate probabilities based on priod knowledge; ie, it enables us to construct conditional probabilities. Mathematically, Bayes' Theorem is $$ P(class|data) = \frac{P(data|class) * P(class)}{P(data)}\equiv \;\frac{\text{prior }\times \text{ likelihood}}{\text{evidence}} $$

where $P(class|data)$ is the probability of a class ($Y=1,2,...$) given available data. The numerator is actually a pretty hard object since it's really the joint probability so there are a lot of terms to interact. This would be a problem when the number of features or observations is large.

This model is called "Naive Bayes" or "Idiot Bayes" because we take a shortcut and assume that the conditional probabilities are independent which simplifies the calculation tremendously. This is a very strong assumption that is most unlikely in real data since it seems obvious that the features will covary. Shockingly, this approach performs surprisingly well even on data where this assumption surely does not hold. Hence, our model seems to be a good approximation.

The last thing to do is decide what distribution $P(\cdot)$ to use. When features are continuous, a common (and reasonable) assumption is that the continuous values associated with each class are distributed according to a normal (ie, Gaussian) distribution. That's what we'll do below but be aware there are other distributions we could have used.

Practice ¶

1. Fit the Naive-Bayes model to the training data; i.e.,

   from sklearn.naive_bayes import GaussianNB
   gnb = GaussianNB()
   gnb.fit(X_train, y_train)
   

1. Use the testing data to identify how well the model performs.

1. Describe the observations which are misclassified. Are there any trends?