Stemming and Lemmatization in Python

Hands-on Stemming and Lemmatization Examples in Python with NLTK

Now you have an overview of stemming and lemmatization. In this section, we are going to get hands-on and demonstrate examples of both techniques using Python and a library called NLTK. 

A brief primer to the Python NLTK package

Natural Language Tool Kit (NLTK) is a Python library used to build programs capable of processing natural language. The library can perform different operations such as tokenizing, stemming, classification, parsing, tagging, semantic reasoning, sentiment analysis, and more.

The latest version is NLTK 3.8.1, and it requires Python versions 3.7, 3.8, 3.9, 3.10, or 3.11, but you don’t have to worry about this since it comes preinstalled in the DataLab workbook – just import nltk and you’re good to go. 

Python Stemming example

One of the most popular stemming algorithms is called the “Porter stemmer.” The porter stemmer was first proposed by Martin Porter in a 1980 paper titled "An algorithm for suffix stripping." The paper has become one of the most common algorithms for stemming in English.

Let’s see how it works:

import nltk
from nltk.stem import PorterStemmer
nltk.download("punkt")
​
# Initialize Python porter stemmer
ps = PorterStemmer()
​
# Example inflections to reduce
example_words = ["program","programming","programer","programs","programmed"]
​
# Perform stemming
print("{0:20}{1:20}".format("--Word--","--Stem--"))
for word in example_words:
   print ("{0:20}{1:20}".format(word, ps.stem(word)))

"""
--Word--            --Stem--            
program             program             
programming         program             
programer           program             
programs            program             
programmed          program

"""

This is a pretty simple example; we expected these results from our porter stemmer as mentioned in the “Stemming” section above. 

Let’s try a trickier example:

import string
from nltk.tokenize import word_tokenize

example_sentence = "Python programmers often tend like programming in python because it's like english. We call people who program in python pythonistas."

# Remove punctuation
example_sentence_no_punct = example_sentence.translate(str.maketrans("", "", string.punctuation))

# Create tokens
word_tokens = word_tokenize(example_sentence_no_punct)

# Perform stemming
print("{0:20}{1:20}".format("--Word--","--Stem--"))
for word in word_tokens:
    print ("{0:20}{1:20}".format(word, ps.stem(word)))

"""
--Word--            --Stem--            
Python              python              
programmers         programm            
often               often               
tend                tend                
like                like                
programming         program             
in                  in                  
python              python              
because             becaus              
its                 it                  
like                like                
english             english             
We                  we                  
call                call                
people              peopl               
who                 who                 
program             program             
in                  in                  
python              python              
pythonistas         pythonista
"""

Here you can see some of the output words are not part of the english dictionary (i.e., “becaus,” “people,” and “programm.”). Another thing to notice is that context is not taken into consideration. For instance, “programmers” is a plural noun but it was reduced down to “program,” which can be a noun or a verb – in other words, the root words are ambiguous. 

Python Lemmatization example

The motivation behind context-sensitive lemmatizers was to improve the performance on unseen and ambiguous words. In our lemmatization example, we will be using a popular lemmatizer called WordNet lemmatizer. 

Wordnet is a large, free, and publicly available lexical database for the English language aiming to establish structured semantic relationships between words.

Let’s see in action:

from nltk.stem import WordNetLemmatizer
nltk.download("wordnet")
nltk.download("omw-1.4")
​
# Initialize wordnet lemmatizer
wnl = WordNetLemmatizer()
​
# Example inflections to reduce
example_words = ["program","programming","programer","programs","programmed"]
​
# Perform lemmatization
print("{0:20}{1:20}".format("--Word--","--Lemma--"))
for word in example_words:
   print ("{0:20}{1:20}".format(word, wnl.lemmatize(word, pos="v")))

"""
--Word--            --Lemma--           
program             program             
programming         program             
programer           programer           
programs            program             
programmed          program
"""   

Input words passed to our lemmatizer will remain unchanged if it cannot be found in WordNet. This means context must be provided, which is done by giving the value for the part-of-speech parameter,  pos, in wordnet_lemmatizer.lemmatize.

Notice the word “programmer” were not cut down to “program” by our lemmatizer: this is because we told our lemmatizer to only stem verbs. 

Let’s pass our lemmatizer some something more complicated to see how it fairs…

example_sentence = "Python programmers often tend like programming in python because it's like english. We call people who program in python pythonistas."
​
# Remove punctuation
example_sentence_no_punct = example_sentence.translate(str.maketrans("", "", string.punctuation))
​
word_tokens = word_tokenize(example_sentence_no_punct)
​
# Perform lemmatization
print("{0:20}{1:20}".format("--Word--","--Lemma--"))
for word in word_tokens:
   print ("{0:20}{1:20}".format(word, wnl.lemmatize(word, pos="v")))
"""
--Word--            --Lemma--           
Python              Python              
programmers         programmers         
often               often               
tend                tend                
like                like                
programming         program             
in                  in                  
python              python              
because             because             
its                 its                 
like                like                
english             english             
We                  We                  
call                call                
people              people              
who                 who                 
program             program             
in                  in                  
python              python              
pythonistas         pythonistas 
"""

All words returned by the lemmatization algorithm is in the english dictionary - minus “pythonistas,” which is more of an informal term used to refer to python programmers. 

Stemming vs Lemmatization

You’ve seen how to implement both techniques, but how do they compare? 

Stemming and lemmatization are both text-processing techniques that aim to reduce inflected words to a common base root. Despite the correlation in the overarching objective, the two techniques are not the same. 

The main differences between stemming and lemmatization lay in how each technique arrives at the objective of reducing inflected words to a common base root. 

Stemming algorithms attempt to find the common base roots of various inflections by cutting off the endings or beginnings of the word. The chop is based on a list of common prefixes and suffixes that can typically be found in inflected words. This non-discriminatory nature act of chopping words may occasionally lead to finding meaningful word stems, but other times it does not. 

On the other hand, lemmatization algorithms attempt to find common base roots from inflected words by conducting a morphological analysis. To accurately reduce inflections, a detailed dictionary must be kept so the algorithm can search through to link an inflected word back to its lemma.

“The two may also differ in that stemming most commonly collapses derivationally related words, whereas lemmatization commonly only collapses the different inflectional forms of a lemma.”

– Source: Stanford NLP, IR Book.

The crude heuristic approach taken by stemming algorithms typically means they’re fast and efficient but not always accurate. In contrast, lemmatization algorithms sacrifice speed and efficiency for accuracy, thus, resulting in meaningful base roots. 

For example, a stemming algorithm may reduce “saw” down to “s.” A lemmatization algorithm will consider whether “saw” is a noun (the hand tool for cutting) or a verb (to see) based on the context in which it is used before deciding to return a lemma – if it’s a noun it will return “saw,” and if it’s a verb it will return “see.”

These points may be pretty clear to you by now, so here’s the million dollar question –  “should I use stemming or lemmatization for text preprocessing?” 

Like most things software related; it depends. 

Do you care about speed and efficiency? If so, choose stemming. 

Is context important for your application? If you said “yes,” then use lemmatization. 

Which technique you use completely depends on the application you are working on and your goals for the project. You may want to run experiments with both techniques and compare the results to see which approach resulted in the outcomes that most align with your project goals. 

Something we have not touched on much in this tutorial is how lemmatization algorithms are created; this is because there are several libraries (such as SpaCy, NLTK, etc.) compatible with different languages. However, if you had to create your own lemmatizer for an unavailable language (i.e., Akan), you would need a good knowledge and understanding of the target language to build a lemmatizer. Stemming algorithms are much easier to build for such scenarios.

Wrap up

To summarize, stemming and lemmatization are techniques used for text processing in NLP. They both aim to reduce inflections down to common base root words, but each takes a different approach in doing so. The stemming approach is much faster than lemmatization but it’s more crude and can occasionally lead to unmeaningful common base roots. Alternatively, lemmatization is much more accurate than stemming in terms of finding meaningful dictionary words, and it takes context into consideration.