Lecture 24: Classification [FINISHED]

Classification is a task that requires the use of machine learning algorithms to learn how to assign a class label to examples from the problem domain. We may want to classify:

• email as "spam" or "not spam".
• a tumor as "benign" or "malignant".
• a flower as a particular species.
• which car a person is likely to buy.

The first two example are binary -- the outcome is either true or not. The second two examples are multi-class -- there are several different discrete values the outcome could take.

There are many different types of classification tasks that you may encounter in machine learning and specialized approaches to modeling that may be used for each. In this lecture we'll touch on some of the more popular:

1. Binary Classification Models
   1.1 Load and Explore Data
   1.2 Logistic
   

1. Multi-Class Classification Models
   2.1 Load and Explore Data
   2.2 Logistic
   

Process
The process for fitting a model is:

(i) Initialize the model by creating a model object.
(ii) Fit the model on training data.
(iii) Test model accuracy by comparing test data predictions with test data results.

Note: Each model has what are called "hyperparameters" which we could "tune" using cross-validation as in the last lecture. For expediency, we won't do that in this lecture but you should in a real-world application.

Class Announcements

PS5 is on eLC. New deadline: Tuesday, November 22nd.

1. Binary Classification Models (top)¶

When the outcome variable is either true (1) or not (0), we can predict the outcome with a binary classification model. Let's load some data and get started.

1.1 Load and Explore Data on Breast Cancer¶

sklearn comes with a bunch of pre-loaded datasets. You can see the available datasets here. Some of these we've already used in their csv form. Let's load the data on breast cancer by using the following function to transform the data into a dataframe. This isn't required to fit our models but let's stick with pandas since it's so powerful.

The data descriptions are here. Let's take a look at the data:

Split the Data into Training and Testing Data Sets¶

1.2 Logistic Regression ¶

The logistic regression passes the linear model through a non-linear function that constrains the output to lie between zero and one. In the logistic case, the function looks like

$$\text{Prob}(Y=1|X) = \frac{\exp \left({\beta_0+\beta_1 X}\right)}{1+\exp \left({\beta_0+\beta_1 X}\right)}.$$

The classifier gives us a set of outputs or classes based on probability when we pass the inputs through the above prediction function and returns a probability score between 0 and 1. We're currently considering the binary case where Y=1 or Y=0. We ecide with a threshold value above which we classify values into Class 1 and of the value goes below the threshold then we classify it in Class 2. Here, we predict the outcome variable is true whenever $\text{Prob}(Y=1|X) \ge 0.5$. Graphically, this amounts to:

When we fit the model to the training data, the algorithm chooses $\beta$ such that:

$$ \hat\beta = \max_\beta \sum_i \Big[y_i\log\big(f(x_i,\beta)\big) + (1-y_i)\log\big(1-f(x_i,\beta)\big)\Big] $$

where

$$ f(x_i,\beta)=\frac{\exp \left({\beta_0+\beta_1 x_i}\right)}{1+\exp \left({\beta_0+\beta_1 x_i}\right)} $$

What's happening here? In the first equation, the optimal $\beta$ generates predictions from $f(X,\beta)$ as consistent as possible with the observed ouctomes (y).

Some things to note:

• The objective function in the first equation is called a log-likelihood since it represents the model's ability the "likelihood" of predicting the outcome (y).
• The computer actually solves for the minimum so replace the $\max$ operator with the $\min$ operator and put a negative in front of the $\sum$ to convert the equation to what the computer is atually doing.
• In the math above there is only one feature in X but of course that's just a simplification. There can be many features!

Using Logistic Regression to Diagnose Cancer¶

Let's see how well the logistic model diagnoses breast cancer.

Wow, that worked really well! Let's try a different model and see if we can do better.

1.3 Testing Results¶

Confusion Matrix¶

We can evaluate how well the model is doing by decomposing the successes and failures using a tool called a "confusion matrix." A confusion matrix is a table which we use describe the performance of a classification model (or "classifier") on a set of test data for which the true values are known.

Let's look at the confusion matrix in our Logit ML:

Diagonal entries are when the model was successful. As for errors:

• 18 times the model predicted the tumor was malignant when it fact it was benign ("False Positive" or "Type I Error")

• 13 times the model predicted the tumor was bening when it fact it was malignant ("False Negative" or "Type II Error")

We can convert the confusion matrix into useful statistics:

• Precision is the ratio tp / (tp + fp) where tp is the number of true positives and fp the number of false positives. The precision is intuitively the ability of the classifier to not label a sample as positive if it is negative (i.e., avoid Type 1 errors).

• Recall is the ratio tp / (tp + fn) where tp is the number of true positives and fn the number of false negatives. The recall is intuitively the ability of the classifier to find all the positive samples.

• F-beta score can be interpreted as a weighted harmonic mean of the precision and recall, where an F-beta score reaches its best value at 1 and worst score at 0.

We can access all of these statistics in the classification_report.

The receiver operating characteristic (ROC) curve is another common tool used with binary classifiers. The dotted line represents the ROC curve of a purely random classifier; a good classifier stays as far away from that line as possible (toward the top-left corner). Area under the ROC Curve (AUC) is a way of translating the figure into a statistic where bigger equals better: A model whose predictions are 100% wrong has an AUC of 0 while a model whose predictions are 100% correct has an AUC of 1.

As you will see, this classifier does better than random guesses.

2. Multi-Class Classification Models (top)¶

Data may be discrete and also non-binary. In this section we discuss models to deal with multiclass discrete outcomes. Note also that any of these can be used in the binary case.

In a previous lecture we used the logit model to classify flower species based on the dimensions of its petal and sepal. Let's start there and then introduce additional tools.

2.1 Load and Explore Data on Flowers (top)¶

2.2 Multiclass Logistic Regression ¶

The Logistic model can be used in multicase contexts as well. Let's fit the model again to the flower data.

Practice ¶

1. Split the data into training and testing data sets. Set the test_size=0.2 and the random_state=10.

1. Fit the logistic model to the training data.

1. Use the testing data to identify how well the model performs.

1. Describe the observations which are misclassified. Are there any trends?