DL - RNN Tensorflow

NLP - Tweet Binary Classification (Disaster tweet or Not)

In this experiment, I tired multiple experiments to build a binary classification model with high accuracy to classify tweets into "Disaster or Not Disaster tweets". For that I started from traditional Machine learning baseline model Naive Bayes to complex Bi-Directional LSTM model and also used pretrained google's Universal Sentence Encoder (USE) model.

Using this I was able to achieve 82% accuracy. All the model that were build were fairly very simple, with one to two hidden layers and multiple of 8 neurons or units. This was done because of the quantity of the data with was not much (deep learning require very large amount of data to obtain high accuracy and to avoid overfitting).

Experiments Details

Following section will cover details about the different experiments like EDA, Models architecture, and its performance.

• Dataset

  The dataset was downloaded from the Kaggle: 'Natural Language Processing with Disaster Tweets' which contained total of 7,613 tweets, in which 4,342 belonged to Not Disaster class tweets and 3,271 belonged to Disaster class tweets. The ratio was fairly balanced (60% Negative Class & 40% positive class) so I considered that Dataset was not skewed.

  Dataset examples:

  Class    ::     " Disaster"
  Text		Australia's Ashes disaster - how the collapse unfolded at Trent Bridge - Telegraph http://t.co/6FYnerMUsG
  Class    ::     "Not Disaster"
  Text		failure is a misfortune but regret is a catastrophe

• Models Architecture

  I have done total of 8 experiments from building baseline model like Naive Bayes, RNN based models, 1D Convolutional NN model, Feed-Forward neural network Dense model and also used pretrained model. I also converted the text into TextVectorization and also trainable and learnable TextEmbeddings. All of these vectorization and embedding layers were also build using `tensorflow.keras.layers.experimental.preprocessing.TextVectorization` and `tensorflow.keras.layers.Embedding` modules. Other details are in the table.

  Models Architecture	Library	Hidden-Layers
  Naive Bayes	Sklearn
  Feed-forward NN	Tensorflow	Vectorization, Embedding, Pooling1D, Dense
  LSTM	Tensorflow	Vectorization, Embedding, LSTM(64), Dense
  GRU	Tensorflow	Vectorization, Embedding, GRU(64), Dense
  Bidirectional-LSTM	Tensorflow	Vectorization, Embedding, Bi-LSTM(64), Dense
  Conv1D NN	Tensorflow	Vectorization, Embedding, Conv1D(64, 5), Pooling1D, Dense
  Pretrained USE	Tensorflow	USE layer, Dense(64), Dense

• Models Performance

  All of the models were trained on few 5 Epochs because dataset was small and to avoid overfitting, but still RNN's based models were overfitting. Please check the code file to understant the overfitting. Perfromance details are:

  Models Architecture	Accuracy
  Naive Bayes	79%
  Feed-forward NN	78%
  LSTM	75%
  GRU	76%
  Bidirectional-LSTM	76%
  Conv1D NN	77%
  Pretrained USE	81%

  And following graphs shows the f1-score of each model experiments..

  

Predictions & Analysis

I also did custom predictions on unseen tweets from the world, for the I used model 6 which was based on transfer learning universal-sentence-encoder USE model. It was gave all the correct class of the tweets.

More experiments can be done to improve models accuracy especially RNN based models which are overfitting with very simple model so more data is required to avoid overfitting and using some model regularization.

to check the most wrong prediction, other classification metrics likes precision or recall please visit my github page and check the code file..

Github page