Lecture 15: Advanced String Search with Regular Expressions [SUGGESTED SOLUTIONS]

We have seen some basic methods for searching text using str.contain() or str.find() and passing strings to match. That works great when we know what we are searching for and the text we are searching has some order to it. In this notebook, we introduce regular expressions. Regular expressions (regex) use a special syntax that allows for powerful searching. For example, a regular expression could find the first four instances of numbers within a string (regardless of their values) or search for two capitalized letters followed by a comma.

Regular expressions are not unique to pandas or python. They were developed as part of the theoretical basis for computing languages and are quite interesting in their own right. They can also confusing.

Our goal is to learn a few regular expressions that might be helpful in the context of processing text data for analysis. Today, we'll work with string functions directly. There is also a specific regex package called re. These skills will be useful for not just cleaning text data but also later when we do classification and prediction with text.

The topic of regular expressions is huge and we will only scratch the surface. If you are interested in learning more, the internet is full of great tutorials. regex101.com is a fantastic sandbox for practicing.

The outline of today's lecture is as follows:

1. Background

1. Milwaukee Call Center Data

1. Regular Expressions

1. Identifying Fire Damage

1. Background (return)¶

So far we've learned how to define and manipulate string objects as well as seen some ways to determine whether two strings match each other; i.e.,

• You can test whether two strings are equal using the equality (==) operator.

• You can test whether one string is a substring of another with the in operator or the built-in string methods .find() and .index().

String matching like this is a common task in programming, and you can get a lot done with string operators and built-in methods. At times, though, you may need more sophisticated pattern-matching capabilities.

Brief History¶

In 1951, mathematician Stephen Cole Kleene described the concept of a 'regular language'; i.e., a language that is recognizable by a finite automaton and formally expressible using regular expressions. In the mid-1960s, computer science pioneer Ken Thompson, one of the original designers of Unix (note that Linux and Apple iOS are unix-based), implemented pattern matching in the QED text editor) using Kleene's notation.

Since then, regexes have appeared in many programming languages, editors, and other tools as a means of determining whether a string matches a specified pattern. Python, Java, and Perl all support regex functionality, as do most Unix/Linux tools and many text editors.

2. Milwaukee Call Center Data (return)¶

The City of Milwaukee open data portal includes data sets on many aspects of the city. Today, we will work with some of the call center data. These data are incidents handled by the city's call center and cover things like missed garbage pick up, litter complaints, and burned out street lights.

We will extract the calls regarding fire damage and compute some statistics about the cost of fires in the city. The estimates of fire damage are embedded in the text description and will need to be extracted.

I've posted the data that I downloaded from the portal so that we are all working from the same dataset but you could download the freshest data, too. Your results would then be a bit different than mine.

Those column names could use some work.

What time span does the data cover? Since we made the 'created' and 'closed' columns datetime, we can use .min() and .max() to find out.

The fire damage calls have the title 'Fire Damage.'

We can now see that there are two damage estimates reports. The first is the damage to the building. The second is the damage to the contents of the building. Some things that might cause difficulties:

1. 'bldg' but also 'bldg.'
2. There are not always 'cont loss' entries.
3. There may be spelling mistakes
4. There maybe inconsistent use of dollar signs or commas

Ugh. There is a reason why people who can do this stuff are paid well.

3. Regular Expressions (return)¶

What follows is a simple overview of regex and some examples to help us understand how they work. You should read through the re package documentation HOWTO (up to "The Backslash Plague") before proceeding.

We split the set of characters into two parts: metacharacters and everything else. Metacharacters have special meanings. The metacharacters are

. ^ $ *  + ? { } [ ] \ | ( )

We will only cover a few of the metacharacters in this notebook.

If we were working in python (and not pandas) we would use the re package. We will use pandas methods such as .findall() and .extract() with regular expressions—pandas already knows how to interpret regular expressions. Many of the string methods in pandas accept either a string or a regex.

Let's take a look at the first text element.

Raw Strings¶

We use the raw string notation which means we do not need to worry about things like \ being interpreted as a special character. Raw strings take the form r'some text', so the only difference between writing a raw string and a regular string is the 'r' in front. Raw strings are actually very useful in programming so they're something to know by themselves.

We put our expression(s) in parentheses. For example, if I want to find the word 'loss' I write

r'(loss)'

The r and apostrophes are for the raw string, the () encapsulate the regex, and the regex itself is loss.

Matching characters¶

Extract occurrences of the word 'loss'.

We'll do this with .findall() which will find all occurrences for the entire column. I am just going to print out a few.

We are given a Series with the extracted text in a list. This is a Series where each element of a column is a list. Returning a list makes is easy to have rows with different numbers of occurrences.

What happens if the extract fails?

Character classes¶

We use [] to specify a set or class of characters to match. Suppose I wanted a a, b, or c.

We can also specify ranges of letter or numbers.

There are some special pre-defined sequences that are used often.

• \w any alphanumeric character: equivalent to [a-zA-Z0-9_]
• \d any numeric digit: equivalent to [0-9_]

the complements are

• \W anything but alphanumeric character: equivalent to [^a-zA-Z0-9_]
• \D anything but numeric digits: equivalent to [^0-9_]

The ^ character is the complement operator. The _ says to look for this character.

Repeating characters¶

• The * means look for the preceding character(s) zero or more times
• The + means look for the preceding character(s) one or more times

The * means that those characters may show up many times or not at all. The + means that the characters have to show up at least once.

Notice the difference between * and +.

• The * expression returns many blank "hits." These are the characters leading up to the first 1. Each one of those characters is not a number, but that is still considered a "hit" because zero decimal characters is considered a match.

• The + expression returns only the clumps of numbers, since characters with zero numbers are excluded. Each clump stops when it hits a nonnumeral: the ,, the ., and the \.

4. Identifying Fire Damage (return)¶

Let's use regex to extract the damages from our data.

My regex is

'\d+,\d+.\d+'

This is looking for at least one number and then all other numbers up to the , then more numbers up to the . and then the rest of the numbers. The + ensures that I am only getting numbers but is general enough to capture both 14 and 193.

Note that this approach could be troublesome if there are other numbers in the text, such as a phone number. We need to always check our results to see if we need to further clean the data. Each dataset is unhappy in its own way.

Did we get everything? Nope. Look at index 539. There is only one entry. The contents loss entry does not have a comma in it so it is not being found. Let's fix this.

Optional characters¶

We use the ? character to say that the preceding character or class is optional. Let's make the comma optional.

We now have caught the contents damage for index 539.

Practice </font >

'bldg loss $14,540.00'