IMDB_GRU_LSTM_Conv1D.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/KoroshRH/IMDB_NLP/blob/main/IMDB_GRU_LSTM_Conv1D.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qnpO3iadYEY2"
      },
      "source": [
        "# Overview\n",
        "\n",
        "In this project, we want to compare different deep learning model architectures on a specific dataset, named [IMDB Reviews dataset](https://www.tensorflow.org/datasets/catalog/imdb_reviews).  \n",
        "Our architecure set comprises 4 types of model: `Flatten`, `LSTM`, `GRU`, and `Conv1D`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "WA0Fi9p9ah5_"
      },
      "outputs": [],
      "source": [
        "import tensorflow_datasets as tfds\n",
        "import tensorflow as tf\n",
        "import numpy as np\n",
        "\n",
        "from tensorflow.keras.preprocessing.text import Tokenizer\n",
        "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
        "import tensorflow as tf"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BTmnR_9dbBY9"
      },
      "source": [
        "## Download and Prepare the Dataset\n",
        "\n",
        "Next, you will download the `plain_text` version of the `IMDB Reviews` dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wHQ2Ko0zl7M4"
      },
      "outputs": [],
      "source": [
        "imdb, info = tfds.load('imdb_reviews', with_info=True, as_supervised=True)\n",
        "\n",
        "train_data, test_data = imdb['train'], imdb['test']\n",
        "\n",
        "training_sentences = []\n",
        "training_labels = []\n",
        "\n",
        "testing_sentences = []\n",
        "testing_labels = []\n",
        "\n",
        "for s,l in train_data:\n",
        "  training_sentences.append(s.numpy().decode('utf8'))\n",
        "  training_labels.append(l.numpy())\n",
        "\n",
        "for s,l in test_data:\n",
        "  testing_sentences.append(s.numpy().decode('utf8'))\n",
        "  testing_labels.append(l.numpy())\n",
        "\n",
        "training_labels_final = np.array(training_labels)\n",
        "testing_labels_final = np.array(testing_labels)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Here we need to set our hyperparameters and do some preprocess methods on our data. These processes are `Tokenizing` and `Indexing`."
      ],
      "metadata": {
        "id": "2Wp0MCVS2Vpi"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7n15yyMdmoH1"
      },
      "outputs": [],
      "source": [
        "vocab_size = 10000\n",
        "max_length = 120\n",
        "trunc_type='post'\n",
        "oov_tok = \"<OOV>\"\n",
        "\n",
        "tokenizer = Tokenizer(num_words = vocab_size, oov_token=oov_tok)\n",
        "\n",
        "tokenizer.fit_on_texts(training_sentences)\n",
        "word_index = tokenizer.word_index\n",
        "\n",
        "sequences = tokenizer.texts_to_sequences(training_sentences)\n",
        "padded = pad_sequences(sequences,maxlen=max_length, truncating=trunc_type)\n",
        "\n",
        "testing_sequences = tokenizer.texts_to_sequences(testing_sentences)\n",
        "testing_padded = pad_sequences(testing_sequences,maxlen=max_length)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cs4GDKAFbJdq"
      },
      "source": [
        "## Plot Helper\n",
        "\n",
        "This method helps us to illustrate our result and compare them to each other easily."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nHGYuU4jPYaj"
      },
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "def plot_graphs(history, string):\n",
        "  plt.plot(history.history[string])\n",
        "  plt.plot(history.history['val_'+string])\n",
        "  plt.xlabel(\"Epochs\")\n",
        "  plt.ylabel(string)\n",
        "  plt.legend([string, 'val_'+string])\n",
        "  plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bUoZJv02bP0m"
      },
      "source": [
        "## Flatten\n",
        "\n",
        "First up is simply using a `Flatten` layer after the embedding. Its main advantage is that it is very fast to train. Observe the results below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_SRAyulSaWAa"
      },
      "outputs": [],
      "source": [
        "embedding_dim = 16\n",
        "dense_dim = 6\n",
        "\n",
        "model_flatten = tf.keras.Sequential([\n",
        "    tf.keras.layers.Embedding(vocab_size, embedding_dim, input_length=max_length),\n",
        "    tf.keras.layers.Flatten(),\n",
        "    tf.keras.layers.Dense(dense_dim, activation='relu'),\n",
        "    tf.keras.layers.Dense(1, activation='sigmoid')\n",
        "])\n",
        "\n",
        "model_flatten.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])\n",
        "\n",
        "model_flatten.summary()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "tYLZUZ3Ga1ok"
      },
      "outputs": [],
      "source": [
        "NUM_EPOCHS = 10\n",
        "BATCH_SIZE = 128\n",
        "\n",
        "history_flatten = model_flatten.fit(padded, training_labels_final, batch_size=BATCH_SIZE, epochs=NUM_EPOCHS, validation_data=(testing_padded, testing_labels_final))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fVPLbqcca6U2"
      },
      "outputs": [],
      "source": [
        "plot_graphs(history_flatten, 'accuracy')\n",
        "plot_graphs(history_flatten, 'loss')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2w_soBeUbSXu"
      },
      "source": [
        "## LSTM\n",
        "\n",
        "Next, you will use an LSTM. This is slower to train but useful in applications where the order of the tokens is important."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wSualgGPPK0S"
      },
      "outputs": [],
      "source": [
        "embedding_dim = 16\n",
        "lstm_dim = 32\n",
        "dense_dim = 6\n",
        "\n",
        "model_lstm = tf.keras.Sequential([\n",
        "    tf.keras.layers.Embedding(vocab_size, embedding_dim, input_length=max_length),\n",
        "    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(lstm_dim)),\n",
        "    tf.keras.layers.Dense(dense_dim, activation='relu'),\n",
        "    tf.keras.layers.Dense(1, activation='sigmoid')\n",
        "])\n",
        "\n",
        "model_lstm.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])\n",
        "\n",
        "model_lstm.summary()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "crEvEcQmUQiL"
      },
      "outputs": [],
      "source": [
        "NUM_EPOCHS = 10\n",
        "BATCH_SIZE = 128\n",
        "\n",
        "history_lstm = model_lstm.fit(padded, training_labels_final, batch_size=BATCH_SIZE, epochs=NUM_EPOCHS, validation_data=(testing_padded, testing_labels_final))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QVwnSYF-aIha"
      },
      "outputs": [],
      "source": [
        "plot_graphs(history_lstm, 'accuracy')\n",
        "plot_graphs(history_lstm, 'loss')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tcBMGJgzcXkl"
      },
      "source": [
        "## GRU\n",
        "\n",
        "The *Gated Recurrent Unit* or [GRU](https://www.tensorflow.org/api_docs/python/tf/keras/layers/GRU) is usually referred to as a simpler version of the LSTM. It can be used in applications where the sequence is important but you want faster results and can sacrifice some accuracy. You will notice in the model summary that it is a bit smaller than the LSTM and it also trains faster by a few seconds."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5NEpdhb8AxID"
      },
      "outputs": [],
      "source": [
        "embedding_dim = 16\n",
        "gru_dim = 32\n",
        "dense_dim = 6\n",
        "\n",
        "model_gru = tf.keras.Sequential([\n",
        "    tf.keras.layers.Embedding(vocab_size, embedding_dim, input_length=max_length),\n",
        "    tf.keras.layers.Bidirectional(tf.keras.layers.GRU(gru_dim)),\n",
        "    tf.keras.layers.Dense(dense_dim, activation='relu'),\n",
        "    tf.keras.layers.Dense(1, activation='sigmoid')\n",
        "])\n",
        "\n",
        "model_gru.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])\n",
        "\n",
        "model_gru.summary()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "V5LLrXC-uNX6"
      },
      "outputs": [],
      "source": [
        "NUM_EPOCHS = 10\n",
        "BATCH_SIZE = 128\n",
        "\n",
        "history_gru = model_gru.fit(padded, training_labels_final, batch_size=BATCH_SIZE, epochs=NUM_EPOCHS, validation_data=(testing_padded, testing_labels_final))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7kwU-2skSQ3E"
      },
      "outputs": [],
      "source": [
        "plot_graphs(history_gru, 'accuracy')\n",
        "plot_graphs(history_gru, 'loss')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ugToQrB-cfr5"
      },
      "source": [
        "## Conv1D\n",
        "\n",
        "Lastly, you will use a convolution layer to extract features from your dataset. You will append a [GlobalAveragePooling1d](https://www.tensorflow.org/api_docs/python/tf/keras/layers/GlobalAveragePooling1D) layer to reduce the results before passing it on to the dense layers. Like the model with `Flatten`, this also trains much faster than the ones using RNN layers like `LSTM` and `GRU`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "K_Jc7cY3Qxke"
      },
      "outputs": [],
      "source": [
        "embedding_dim = 16\n",
        "filters = 128\n",
        "kernel_size = 5\n",
        "dense_dim = 6\n",
        "\n",
        "model_conv = tf.keras.Sequential([\n",
        "    tf.keras.layers.Embedding(vocab_size, embedding_dim, input_length=max_length),\n",
        "    tf.keras.layers.Conv1D(filters, kernel_size, activation='relu'),\n",
        "    tf.keras.layers.GlobalAveragePooling1D(),\n",
        "    tf.keras.layers.Dense(dense_dim, activation='relu'),\n",
        "    tf.keras.layers.Dense(1, activation='sigmoid')\n",
        "])\n",
        "\n",
        "model_conv.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy'])\n",
        "\n",
        "model_conv.summary()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aUV70isnTiFF"
      },
      "outputs": [],
      "source": [
        "NUM_EPOCHS = 10\n",
        "BATCH_SIZE = 128\n",
        "\n",
        "history_conv = model_conv.fit(padded, training_labels_final, batch_size=BATCH_SIZE, epochs=NUM_EPOCHS, validation_data=(testing_padded, testing_labels_final))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "T42EmhV0XhRV"
      },
      "outputs": [],
      "source": [
        "plot_graphs(history_conv, 'accuracy')\n",
        "plot_graphs(history_conv, 'loss')"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "name": "IMDB_GRU_LSTM_Conv1D.ipynb",
      "private_outputs": true,
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.4"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}