Analyzing data that comes in tables is fun. But what if the things that we find <em>most interesting</em> are not available as a neatly organized dataset but in plain text? Do not despair: In this course, you'll learn everything you need to know to <em>create powerful regular expressions</em> that will help you find all the information you need for your analyses from just a blob of text. But not only that. Using the concept of <em>string distances</em> you will learn to work even with text that contains typos or scanning errors, as you will be able to match them to their correct counterparts from other data sources (record linkage). As a learning material, we will analyze real documents about box office figures in Swiss cinemas.
Regular Expressions: Writing custom patternsFree
Regular expressions can be pretty intimidating at first as they contain vast amounts of special characters. In this chapter, you'll learn to decipher these and write your own patterns to find exactly what you're looking for.
Creating strings with data
In this chapter, we will slightly move away from regular expressions and focus on string manipulation by creating strings from other data structures like vectors or lists.
Extracting structured data from text
One task where regular expressions really shine is making sense from a blob of text. In this chapter, you'll learn to extract the information from messy data that doesn't come in neatly arranged tables but in plain text.
Similarities between strings
In the last chapter, we will shift gears away from regular expressions to understanding string distances. By calculating the differences of multiple strings, we can match those that are similar. This will help us to find duplicates even when they contain small errors like typos. This is an important part to record linkage where we combine datasets from multiple sources.
Angelo Zehr is working as a data journalist at SRF, the Swiss Public Broadcaster. In his work, he is regularly confronted with large amounts of messy text that he needs to search and make sense of. In addition to his work, he teaches data journalism at the University of Applied Sciences in Chur and other courses at the Swiss School of Journalism.