Block 102: Stemming & Lemmatization
Reduce words to their base forms.
Concepts
- PorterStemmer and SnowballStemmer from NLTK
- WordNetLemmatizer from NLTK
- Difference: stemming (rules) vs lemmatization (vocabulary)
- POS-aware lemmatization
Code Examples
See exercise below.
Exercise
Compare stemming vs lemmatization for: running, flies, better, geese, studies. Write a function that lemmatizes a sentence using POS tags.
Homework
Describe a scenario where stemming would produce incorrect results but lemmatization would not. Tuesday