Published by FirstAlign
Use of hate speech and offensive language on social media is a growing challenge. This can effect the mental health of a person against whom the hate speech or offensive language is being expressed, with sometimes devastating effects. In order solve or help solving this problem, text classification can be used, so a message on social media can be tagged as hate speech or offensive language.
Text classification is a method in which we assign a class, tag or label to text based on the content. For example tagging an email as ‘spam’ based on the content it is containing. Text classification is a huge field, but one of the fundamental and basic operations of Natural Language Processing (NLP). Because data as text is unstructured, most of the time, in order to make sense of that data we need to perform NLP analysis, which we will discuss each in detail. In our use case we are attempt to perform classification of tweets, whether the content of tweet is considered to be hate speech, offensive language or neither. The following is an outline of the process we undertook.
In order to perform the classification we required a dataset on which to train our classification model. The dataset we are using is a dataset used in Automated Hate Speech Detection and the Problem of Offensive Language. This dataset was collected from twitter, with tweets containing word/ terms forming the hatebase.org lexicon, and then a labeled sample of 25,000 tweets. This dataset contained 3 labels. The first label is 0 which means the tweet is a hate speech. The content of tweet are supposed to insult, humiliate or degrade an individual or group of individuals or a community. The next label is 1 which means the content of tweets contain some offensive language but doesn’t meet the definition of hate speech. The last label is 2, which means the content of tweet is neither hate speech nor contains offensive language.
The figure below shows the frequency of tweets in each class.
Pre-processing and NLP Operations
Now the data set is established it’s time to perform pre-processing and apply NLP. The first task performed was to apply a Regular Expression. This helped us remove all the punctuation and symbols so that we are only left with text. All other characters except for the alphabet add noise to the data and decrease the performance of the model. We applied this step by using ‘re’ python package, leaving only the text.
The second task ws the application of NLP. In order to apply NLP operation we are used the nltk package. Firstly we targeted the removal of stop-words. Stop-words are high frequency and low importance features like the word “the”. It occurs a lot but contributes very little or doesn’t contribute at all in the classification, so we need to remove them for that we have used nltk’s stopword corpus. Secondly we applied the process of Stemming. In this step we striped the actual word to its root. This remove extraneous letter from a word, i.e. the word “quickly” is reduced to its root “quick”, we do this in order to remove the ambiguity and was done with the help of PorterStemmer from the nltk.stem.porter package.
Now we have cleaned the data it can be seen in human readable form, but for computer to make sense of that datset we need to create a matrix of features using a CounterVectorizer..
3. Apply CountVectorizer
In this step we are create a matrix of features in our code. We are take 3600 top features and apply the CountVectorizer to our corpus. This allows us to create a matrix which contain the features, the number of occurrences, creating a matrix with 3600 columns and 24783 rows. To do this we utilized CountVectorizer from sklearn. We now have fully processes dataset that can be fed into the algorithms but before doing that we perform a splitting of the dataset.
4. Splitting the dataset
Now we have data ready to be fed to algorithm, but before doing this we must split our data into two differetnt sets, a training and testing set. We can then evaluate the model. In this case we have used 85% of data for training, and 15% for testing. We achieved it with help of the train_test_split from sklearn python package.
5. Applying Machine Learning Algorithm
Now we have data spliced into train and test sets we will use the training set to train the model with particular algorithms, for this case we have used 4 algorithms;
- Decision Tree
- Random Forest
- Logistic Regression
5.1. Apply Decision Tree Algorithm
This algorithm uses observations and iterations from one observation to another to get the predicted value. Here observation refers to the feature in a model. We used the DecisionTreeClassifier classifier from sklearn python package, and fit this to the training data to get a classifier. The Decision Tree Algorithm then created a prediction vector on the test data we used in the evaluation.
5.2. Apply KNN Algorithm
This algorithm checks that a data point is close to a label based on the training data, checks the number of neighbors and predicts the label of this new data. We used the KNN classifier from sklearn python package, fit this to the training data to get a classifier. The KNN algorithm created a prediction vector on the test data which we used in the evaluation.
5.3. Apply Random Forest Algorithm
This algorithm creates a number of Decision Trees at the time of training, and predicts the output label mode of the classes, or the mean of the prediction is used. This then edits the label of this new data. The Random Forest classifier from sklearn python package, fit to the training data to get a classifier which and a prediction vector created on the test data which we used in the evaluation.
5.4. Apply Logistic Regression Algorithm
This algorithm uses a probability for a certain class/ label to predict the label or class for the given data point. We used a Logistic Regression Algorithm classifier from sklearn python package, fit this to the training data to get a classifier and create a prediction vector on the test data which we used in the evaluation.
6. Evaluation of our models
We have used 4 parameters for evaluation confusion matrix, accuracy precision and recall, for that we have used confusion_matrix, accuracy_score, precision_score and recall_score from sklearn python package.
6.1 Evaluate Decision Tree model
The Decision Tree model has accuracy of 87.9%, precision of 88.7% and a recall score of 87.9%. The confusion matrix for this model shows that 79 values of hate speech were predicted correctly, 112 were classified wrongly as offensive language, and 14 were classified incorrectly as neither. 139 values for offensive language were incorrectly identified as hate speech, 2668 were correctly classified as offensive language, and 74 were wrongly classified as neither. 33 tweets were incorrectly identified as hate speech, when they were which were actually neither, and 75 tweets which were also neither, were predicted as offensive language, with 524 were correctly identified as neither.
6.2 Evaluate KNN model
The confusion matrix for KNN models shows that 42 values were correctly predicted as hate speech. 148 values of hate speech were wrongly classified as offensive language, and 15 hate speech tweets were predicted incorrectly as neither. 62 values of offensive language tweets were classified wrongly as hate speech, 2741 were correctly predicted as offensive language tweets, and 78 tweets from offensive language we predicted as neither. 20 and 224 tweets from neither class were predicted as hate speech or offensive language respectively, and 388 were predict correctly as neither. The accuracy precision recall score for KNN is 85.2%, 83.07% and 85.28% respectively.
6.3 Evaluate Random Forest Model
The accuracy, precision and recall score for Random Score model is 89.05%, 88.347% and 89.05% respectively. The confusion matrix for Random Forest shows that the 61 tweets were correctly predicted as hate speech, 130 hate speech tweets were classified as offensive language, and 14 as neither. 85 of the offensive language tweets were identified as hate speech, 2720 were correctly predicted as offensive language tweets, and 76 of of offensive language tweets were wrongly predicted as neither. 12 tweets were actually labeled as neither, and 90 were predicted as hate speech or offensive language. 530 correct predictions were made that identified as neither.
6.4 Evaluate Logistic Regression Model
The confusion matrix for Logistic Regression models shows 57 and 35 hate speech tweets were classified as offensive language tweet and neither respectively, and 113 tweets were classified correctly as hate speech. 1385 of the offensive language tweets were classified as hate speech and 798 were correctly predicted as offensive language tweet, 698 tweets belonging to offensive language tweets were wrongly predicted as neither. 171 and 64 of tweets which belonged to neither class were predicted hate speech and offensive language tweets respectively, and 397 correct predictions were made for neither class. 90.03%, 89.16% and 90.03% are the accuracy, precision and recall score for logistic regression models.
Use of hate speech or offensive language against an individual, group or community on social media of one the biggest challenges sites such as twitter and facebook are facing. In order to solve this problem text classification can be used. In this article we have presented a case for how text classification can be used. We have applied different machine learning algorithms for prediction, and evaluated each one. To achieves this we have used jupyter notebook and pandas, nltk and sklearn python packages.
From the evaluation it is clear that best algorithm used was Logistic Regression. It provided the best results while KNN provided the worst. One observation is that from confusion matrix, it can be seen that all models except for Logistic Regression were making more incorrect predictions about hate speech. Logistic Regression however, provided better results for hate speech, but for offensive language it was less accurate. The overall performance of Logistic Regression was better than the other three algorithms used.
Hope you enjoyed the article the code and data for article can be found on this GitHub repository
Happy Coding ❤