Imagine back to your school days studying for an exam. Did you randomly read sections of your notes, or randomly do problems in the back of the book? No! Well, at least I hope you didn’t approach your schooling with the same level of rigor as what to eat for breakfast. What you probably did was figure out what topics were difficult for you to master and worked diligently at those. Only doing minor refreshing of ideas that you felt you understood. So why do we treat our machine students differently?
We need more data! It is a clarion call I often hear working as a Data Scientist, and it’s true most of the time. The way this normally happens is some problem doesn’t have enough data to get good results. A manager asks how much data you need. You say more. They hire some interns or go crowdsource some labelers, spend a few thousand dollars and you squeak out a bit more performance. Adding in a single step where you let your model tell you what it wants to learn more about can vastly increase your performance with a fraction of the data and cost. I’m talking about doing some, get ready for the buzz word, active learning.
In this article, we will run some basic experiments related to active learning and data selection. We will train a random forest on a small subset of the IMDB Sentiment dataset. Then we will increase the training set by sampling randomly, and by sampling data points that the model wants to learn about. We will compare our performance increase with respect to increasing data and show how smart labeling can save time, money, and increase performance. The code for this project is in a gist here, and also included at the bottom of this article. Let’s get started.
Appeal to Reader
If you pay for Medium, or haven’t used your free articles for this month, please consider reading this article there. I post all of my articles here for free so everyone can access them, but I also like beer and Medium is a good way to collect some beer money : ). So please consider buying me a beer by reading this article on Medium.
If your problem needs more data, try labeling it with the help of your classifier. Do this by either choosing the examples with the least confidence or the examples where the highest and second-highest probabilities are closest. This works most of the time but is no panacea. I’ve seen random sampling do as well as these active learning approaches.
For this problem, we will be looking at the IMDB Sentiment dataset and trying to predict the sentiment of a movie review. We are going to take the whole test set for this dataset, and a tiny subset of training data. We’ll gradually increase the training set size based on different sampling strategies and look at our performance increase.
There are about 34,881 examples in the training set and only 15,119 in the test set. We start by loading the data into a pandas data frames.
For this tutorial, we’ll look at a simple Random Forest. You can apply these techniques to just about any model you can imagine. The model only needs a way of telling you how confident it is in any given prediction. Since we’re working with text data our basic model will use TF-IDF features from the raw text. I know, I know, we should use a deep transformer model here, but this is a tutorial on active learning not on SOTA so forgive me. If you want to see how to use something like BERT check out my other tutorial here.
We’ll define our RandomForest model as a SciKit-Learn pipeline using only unigram features:
Now we can call .fit() on a list of text input and the pipeline will handle the rest. Let’s use our initial training set of 5 examples and see how we do on the test set.
From this, we can see the dataset is pretty balanced since predicting all positive gives us almost .5 precision. This model is pretty crappy though since it only predicts positive. Let’s see if we can use active learning to get to better performance faster than randomly sampling new points.
Choosing Good Data Points to Label
So we have a classifier now. It’s meh at best, and we want more data. Let’s use the classifier to make predictions on our other training data and see which points the model is least confident about. For most Sci-Kit Learn estimators this is super easy. We can use the .predict_proba() function to get a probability for each class. To do this by hand you could also look at the individual predictions of the trees and count the votes for each class. However, predict_proba is much more convenient : ).
This will give us a numpy array of probabilities where each column is a class and each row is an example. It’s something like:
[[.1, .9], [.5, .5], [.2, .8]...
The simplest “intelligent” strategy for picking good points to label is to use the points which the model is least confident about. In the example above that would be the second point, because the maximum probability of any class is the smallest.
Here we have a function that takes a data frame of training examples, the associated predicted probabilities, and the number of points we want to sample. It then gets the maximum value in each row, sorts the data points from smallest to largest, and grabs the n examples that had the smallest maximum probability.
If we apply uncertainty sampling to the three example probabilities above we’d say we should label [.5, .5] first because the maximum probability is smaller than all the other maximum probabilities. (.8 and .9) which intuitively makes sense!
Uncertainty sampling is nice, but in the multiclass setting, it doesn’t do as good of a job of capturing uncertainty. What if you had the following predictions?
[[.01, .45, .46], [.28, .28, .44], [0.2, 0.0, .80]...
The data point which the model seems to be most uncertain about is the first one since it’s predicting class 3 by just .01! But Uncertainty sampling would say that example two is the best point to label since .44 is the smallest maximum probability. They are both good candidates, but the first intuitively makes more sense. Margin sampling caters to this intuition; that the best points to label are those with the smallest margin between predictions. We can perform margin sampling with the following function:
In this code, we sort the predicted probabilities. Then we check to see if the array has more than one dimension. If it has only one probability it only saw one class and has no information about the existence of other classes. In this case, we need to just randomly sample. Otherwise, we find the margin by subtracting the second-highest probability from the highest probability and sorting the results.
I ran a simple experiment where I start with five randomly sampled points, then apply each sampling strategy to gain five more points. I do this iteratively 100 times until I’ve sampled about 500 points. I plot the f1 score on the test set at each time point and look at how our performance improves with each sampling strategy.
You can see that Marginal Sampling and Uncertainty sampling both do better than random for this problem. They are the same in the binary classification case I didn’t think about this when I started writing this article 😅. I created an additional sampling strategy called combined which does a little bit of margin sampling, a little bit of uncertainty sampling, and a little bit of random sampling. I like this combined approach for many of my projects because sometimes random sampling does help. If we are always sampling according to the margin, or uncertainty, we aren’t sampling uniformly from our dataset and could be missing out on some important information. Anecdotally, I’ve seen a little random in the sampling usually pays off. Though don’t believe me because I haven’t run any good experiments to prove this yet 😅.
Active learning can help you get better performance with fewer data by choosing new points that add the most information to your model. This strategy works pretty well most of the time but isn’t guaranteed to do better. It’s a nice tool to keep in mind when you’re thinking about labeling some additional data.