Transcript:
Now that we know about limitization, it’s time to use limitations present in an ltk package, In particular, we will be using word limitizer so wordnet is a collection of valves adjectives, nouns, adverbs, and these are grouped together on synonyms of these words, so in order to use word net limitizer in Nltk. First, you need to import the nltk package and then an ltk dot word limitizer and we will create an object of this, and then we will use the limitized function that is present inside this, and then we can call it on some words like wn dot limit eyes and then let’s say coder coding and these things. So its syntax is very similar to a portrait stemmer that we have seen in the last video so there we used an ltk dot port restrainer, so in this video, also we will, uh, import both of those and try to compare what’s the difference in results when we use limitation versus stemming. So let’s begin in our notebook. So first, let’s import the nltk package. And then the wordnet limitizer from an ltk. Let’s also import that portrait stemmer in order to compare the results between stemming and limitization and let’s quickly have a look at what are the available functions inside this limitizer so let’s run it. So these are the main functions, and we will be mainly concerned with this limit eyes function, so let’s compare the results of lemmatizer and stemmer on some of the words, so let’s try on booze and, geez, so first, let’s try on a stemmer goose and then geez, so we see that it’s not able to identify that these two belong to same word, so it chops them to GWS and G-w-e-s, so these are not even words in the dictionary, so they don’t make too much of sense. Now, let’s apply the same words and give them to this word net limitizer and let’s run it so here. We see that it’s correctly able to differentiate between these two, and it reduces them to a common word goose and similarly lets [Music] limitize cactus and also cacti and run it, so it’s able to understand that these two belong to the same Lima and it’s able to convert both of them to cactus, but if we run the same example on porter’s Timmer and run it, it’s not able to identify that, and it just blindly chops out this s. So that’s why we see that, uh, this lemmatizer is much more powerful than, uh, stemming so stemming, uses just heuristics and is only concerned with the string it is given, and it essentially chops off some suffix from that word, whereas limitation searches the corpus to find related words and reduce it down to the root word or the Lima. Now, uh, let’s run this, Uh, lemmatizer on our, uh, spam collection data set spam detection dataset, so we will first read the text raw text so we will need to import pandas as PD. So this is just the old stuff and then re, and we will also import string for punctuation, removing punctuation, and then we will set option display Dot Max call width 200 and we will save the stock words for English language you and let’s print the first five rows to see everything is working fine. So there is some error messages. Name MSG is not defined. Okay, so this should be string. Yes, so till now, it’s fine, and this is just the old stuff. We are reading our data from SMS spam collection and separating them based on tab, and we have named the two columns in this data frame as label and message, so this label contains hammer spam and this MSG contains the actual message. Now we will clean the text. Now we will define our own custom function to clean the text so first we will get rid of the punctuation, then we will split it into tokens and we will use re Dot split and split it on non-word characters, and then finally we will get rid of those top words, and then we will return the text and then we will use this clean function and create a new column to store the list of words without stop words and punctuations, so let’s name it MSG note text, no, no stop. And then we will apply Lambda function on this MSG column and let’s print the first five rows to see if it’s working correctly, so we see that it’s working correctly and it could have removed some of the stop words like here until is removed, so it’s not there similarly here in is removed so only is removed in is removed, so we can see that many of the stop words are removed, so now we are ready to apply the lemmatizer on the SMS spam collection data, so let’s define a limitation function and it will take the list of tokenized words so again we will use the same thing, but instead of returning word, we will apply that, uh, wordnet limitizer on that, so we will write the blue and or not limit eyes and here tokenize text, and then we can return this text. Now let’s create a new column and we will call it MSG limitized and we will apply Lambda function on this last column for each word, we will limitize them and let’s print the first five rows and see the results. Uh, so we are not seeing much of effect here, so here we see that goes is converted to go and lives is converted to life, so these are not proper English words, so we are not seeing too much of effect, but we see that it’s on some words we can see the effect and here on I is converted, it’s O only so so we are seeing very less effect because these are not proper English words, so that’s how we lemmetize our texts See you in the next video, where we will, uh, move to the next stage of NLP pipeline where we will vectorize, uh, our texts into numbers which can be consumed by our machine learning algorithms. So see you in the next video, thank you.