19/rl-baselines-zoo.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "rl-baselines-zoo.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/decoderkurt/HUF_RL_2022/blob/main/19/rl-baselines-zoo.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XJy9QoDC7XA7"
      },
      "source": [
        "# RL Baselines3 Zoo: Training in Colab\n",
        "\n",
        "\n",
        "\n",
        "Github Repo: [https://github.com/DLR-RM/rl-baselines3-zoo](https://github.com/DLR-RM/rl-baselines3-zoo)\n",
        "\n",
        "Stable-Baselines3 Repo: [https://github.com/DLR-RM/rl-baselines3-zoo](https://github.com/DLR-RM/stable-baselines3)\n",
        "\n",
        "\n",
        "# Install Dependencies\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AXVDDlTn02M9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "31f8ff72-8fdb-49a4-a30b-dc60a3a232c2"
      },
      "source": [
        "!apt-get install swig cmake ffmpeg freeglut3-dev xvfb"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Reading package lists... Done\n",
            "Building dependency tree       \n",
            "Reading state information... Done\n",
            "freeglut3-dev is already the newest version (2.8.1-3).\n",
            "swig is already the newest version (3.0.12-1).\n",
            "cmake is already the newest version (3.10.2-1ubuntu2.18.04.2).\n",
            "ffmpeg is already the newest version (7:3.4.8-0ubuntu0.2).\n",
            "xvfb is already the newest version (2:1.19.6-1ubuntu4.10).\n",
            "0 upgraded, 0 newly installed, 0 to remove and 37 not upgraded.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kDjF3qRg7oGH"
      },
      "source": [
        "## Clone RL Baselines3 Zoo Repo"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SCjGikdT1DFy",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "95edddd5-81c2-4d0a-b137-3cd192eeadbb"
      },
      "source": [
        "!git clone --recursive https://github.com/DLR-RM/rl-baselines3-zoo"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "fatal: destination path 'rl-baselines3-zoo' already exists and is not an empty directory.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "REMQlh-ezyVt",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a0c607d4-5f3d-4c6f-bd6f-e2b958ca32a1"
      },
      "source": [
        "%cd /content/rl-baselines3-zoo/"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/rl-baselines3-zoo\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5tmD_QTBqTMb"
      },
      "source": [
        "### Install pip dependencies"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OWIDzgJTqShY",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "555add5b-679f-4d71-ef4e-57c9864257fb"
      },
      "source": [
        "!pip install -r requirements.txt"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: gym<0.20,>=0.17 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 1)) (0.17.3)\n",
            "Requirement already satisfied: stable-baselines3[docs,extra,tests]>=1.3.1a8 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 2)) (1.3.1a9)\n",
            "Requirement already satisfied: sb3-contrib>=1.3.1a7 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 3)) (1.3.1a7)\n",
            "Requirement already satisfied: box2d-py==2.3.8 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 4)) (2.3.8)\n",
            "Requirement already satisfied: pybullet in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 5)) (3.2.1)\n",
            "Requirement already satisfied: gym-minigrid in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 6)) (1.0.2)\n",
            "Requirement already satisfied: scikit-optimize in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 7)) (0.9.0)\n",
            "Requirement already satisfied: optuna in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 8)) (2.10.0)\n",
            "Requirement already satisfied: pytablewriter in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 9)) (0.64.1)\n",
            "Requirement already satisfied: seaborn in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 10)) (0.11.2)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 11)) (6.0)\n",
            "Requirement already satisfied: cloudpickle>=1.5.0 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 12)) (1.6.0)\n",
            "Requirement already satisfied: atari-py==0.2.6 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 14)) (0.2.6)\n",
            "Requirement already satisfied: plotly in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 15)) (4.4.1)\n",
            "Requirement already satisfied: panda-gym==1.1.1 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 16)) (1.1.1)\n",
            "Requirement already satisfied: rliable>=1.0.5 in /usr/local/lib/python3.7/dist-packages (from -r requirements.txt (line 17)) (1.0.5)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from atari-py==0.2.6->-r requirements.txt (line 14)) (1.19.5)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from atari-py==0.2.6->-r requirements.txt (line 14)) (1.15.0)\n",
            "Requirement already satisfied: pyglet<=1.5.0,>=1.4.0 in /usr/local/lib/python3.7/dist-packages (from gym<0.20,>=0.17->-r requirements.txt (line 1)) (1.5.0)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from gym<0.20,>=0.17->-r requirements.txt (line 1)) (1.7.3)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (1.1.5)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (3.2.2)\n",
            "Requirement already satisfied: torch>=1.8.1 in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (1.10.0+cu111)\n",
            "Requirement already satisfied: absl-py>=0.9.0 in /usr/local/lib/python3.7/dist-packages (from rliable>=1.0.5->-r requirements.txt (line 17)) (0.12.0)\n",
            "Requirement already satisfied: arch>=4.19 in /usr/local/lib/python3.7/dist-packages (from rliable>=1.0.5->-r requirements.txt (line 17)) (5.1.0)\n",
            "Requirement already satisfied: property-cached>=1.6.4 in /usr/local/lib/python3.7/dist-packages (from arch>=4.19->rliable>=1.0.5->-r requirements.txt (line 17)) (1.6.4)\n",
            "Requirement already satisfied: statsmodels>=0.11 in /usr/local/lib/python3.7/dist-packages (from arch>=4.19->rliable>=1.0.5->-r requirements.txt (line 17)) (0.13.1)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (2018.9)\n",
            "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/dist-packages (from pandas->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (2.8.2)\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from pyglet<=1.5.0,>=1.4.0->gym<0.20,>=0.17->-r requirements.txt (line 1)) (0.16.0)\n",
            "Requirement already satisfied: pillow in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (7.1.2)\n",
            "Requirement already satisfied: tensorboard>=2.2.0 in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (2.7.0)\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (5.4.8)\n",
            "Requirement already satisfied: opencv-python in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (4.1.2.30)\n",
            "Requirement already satisfied: flake8-bugbear in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (22.1.11)\n",
            "Requirement already satisfied: pytest in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (4.3.1)\n",
            "Requirement already satisfied: pytest-env in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (0.6.2)\n",
            "Requirement already satisfied: flake8>=3.8 in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (4.0.1)\n",
            "Requirement already satisfied: isort>=5.0 in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (5.10.1)\n",
            "Requirement already satisfied: pytest-xdist in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (1.27.0)\n",
            "Requirement already satisfied: black in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (21.12b0)\n",
            "Requirement already satisfied: pytest-cov in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (2.9.0)\n",
            "Requirement already satisfied: pytype in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (2022.1.13)\n",
            "Requirement already satisfied: sphinx-rtd-theme in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (1.0.0)\n",
            "Requirement already satisfied: sphinx-autodoc-typehints in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (1.15.2)\n",
            "Requirement already satisfied: sphinx in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (4.3.2)\n",
            "Requirement already satisfied: sphinxcontrib.spelling in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (7.3.2)\n",
            "Requirement already satisfied: sphinx-autobuild in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (2021.3.14)\n",
            "Requirement already satisfied: pycodestyle<2.9.0,>=2.8.0 in /usr/local/lib/python3.7/dist-packages (from flake8>=3.8->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (2.8.0)\n",
            "Requirement already satisfied: mccabe<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from flake8>=3.8->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (0.6.1)\n",
            "Requirement already satisfied: pyflakes<2.5.0,>=2.4.0 in /usr/local/lib/python3.7/dist-packages (from flake8>=3.8->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (2.4.0)\n",
            "Requirement already satisfied: importlib-metadata<4.3 in /usr/local/lib/python3.7/dist-packages (from flake8>=3.8->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (4.2.0)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata<4.3->flake8>=3.8->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (3.7.0)\n",
            "Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata<4.3->flake8>=3.8->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (3.10.0.2)\n",
            "Requirement already satisfied: patsy>=0.5.2 in /usr/local/lib/python3.7/dist-packages (from statsmodels>=0.11->arch>=4.19->rliable>=1.0.5->-r requirements.txt (line 17)) (0.5.2)\n",
            "Requirement already satisfied: protobuf>=3.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (3.17.3)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (57.4.0)\n",
            "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (1.8.1)\n",
            "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (0.37.1)\n",
            "Requirement already satisfied: grpcio>=1.24.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (1.43.0)\n",
            "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (0.6.1)\n",
            "Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (1.35.0)\n",
            "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (0.4.6)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (3.3.4)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (1.0.1)\n",
            "Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (2.23.0)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.2.0->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (0.2.8)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.2.0->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (4.8)\n",
            "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.2.0->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (4.2.4)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=2.2.0->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (1.3.0)\n",
            "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard>=2.2.0->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (0.4.8)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard>=2.2.0->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (3.0.4)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard>=2.2.0->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard>=2.2.0->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (2.10)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard>=2.2.0->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (2021.10.8)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=2.2.0->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (3.1.1)\n",
            "Requirement already satisfied: scikit-learn>=0.20.0 in /usr/local/lib/python3.7/dist-packages (from scikit-optimize->-r requirements.txt (line 7)) (1.0.2)\n",
            "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-optimize->-r requirements.txt (line 7)) (1.1.0)\n",
            "Requirement already satisfied: pyaml>=16.9 in /usr/local/lib/python3.7/dist-packages (from scikit-optimize->-r requirements.txt (line 7)) (21.10.1)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn>=0.20.0->scikit-optimize->-r requirements.txt (line 7)) (3.0.0)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from optuna->-r requirements.txt (line 8)) (4.62.3)\n",
            "Requirement already satisfied: sqlalchemy>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from optuna->-r requirements.txt (line 8)) (1.4.29)\n",
            "Requirement already satisfied: cmaes>=0.8.2 in /usr/local/lib/python3.7/dist-packages (from optuna->-r requirements.txt (line 8)) (0.8.2)\n",
            "Requirement already satisfied: cliff in /usr/local/lib/python3.7/dist-packages (from optuna->-r requirements.txt (line 8)) (3.10.0)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.7/dist-packages (from optuna->-r requirements.txt (line 8)) (21.3)\n",
            "Requirement already satisfied: colorlog in /usr/local/lib/python3.7/dist-packages (from optuna->-r requirements.txt (line 8)) (6.6.0)\n",
            "Requirement already satisfied: alembic in /usr/local/lib/python3.7/dist-packages (from optuna->-r requirements.txt (line 8)) (1.7.5)\n",
            "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging>=20.0->optuna->-r requirements.txt (line 8)) (3.0.6)\n",
            "Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.7/dist-packages (from sqlalchemy>=1.1.0->optuna->-r requirements.txt (line 8)) (1.1.2)\n",
            "Requirement already satisfied: tcolorpy<1,>=0.0.5 in /usr/local/lib/python3.7/dist-packages (from pytablewriter->-r requirements.txt (line 9)) (0.1.1)\n",
            "Requirement already satisfied: pathvalidate<3,>=2.3.0 in /usr/local/lib/python3.7/dist-packages (from pytablewriter->-r requirements.txt (line 9)) (2.5.0)\n",
            "Requirement already satisfied: DataProperty<2,>=0.54.2 in /usr/local/lib/python3.7/dist-packages (from pytablewriter->-r requirements.txt (line 9)) (0.54.2)\n",
            "Requirement already satisfied: tabledata<2,>=1.3.0 in /usr/local/lib/python3.7/dist-packages (from pytablewriter->-r requirements.txt (line 9)) (1.3.0)\n",
            "Requirement already satisfied: mbstrdecoder<2,>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from pytablewriter->-r requirements.txt (line 9)) (1.1.0)\n",
            "Requirement already satisfied: typepy[datetime]<2,>=1.2.0 in /usr/local/lib/python3.7/dist-packages (from pytablewriter->-r requirements.txt (line 9)) (1.3.0)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (0.11.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (1.3.2)\n",
            "Requirement already satisfied: retrying>=1.3.3 in /usr/local/lib/python3.7/dist-packages (from plotly->-r requirements.txt (line 15)) (1.3.3)\n",
            "Requirement already satisfied: Mako in /usr/local/lib/python3.7/dist-packages (from alembic->optuna->-r requirements.txt (line 8)) (1.1.6)\n",
            "Requirement already satisfied: importlib-resources in /usr/local/lib/python3.7/dist-packages (from alembic->optuna->-r requirements.txt (line 8)) (5.4.0)\n",
            "Requirement already satisfied: typed-ast>=1.4.2 in /usr/local/lib/python3.7/dist-packages (from black->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (1.5.1)\n",
            "Requirement already satisfied: click>=7.1.2 in /usr/local/lib/python3.7/dist-packages (from black->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (7.1.2)\n",
            "Requirement already satisfied: platformdirs>=2 in /usr/local/lib/python3.7/dist-packages (from black->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (2.4.1)\n",
            "Requirement already satisfied: tomli<2.0.0,>=0.2.6 in /usr/local/lib/python3.7/dist-packages (from black->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (1.2.3)\n",
            "Requirement already satisfied: pathspec<1,>=0.9.0 in /usr/local/lib/python3.7/dist-packages (from black->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (0.9.0)\n",
            "Requirement already satisfied: mypy-extensions>=0.4.3 in /usr/local/lib/python3.7/dist-packages (from black->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (0.4.3)\n",
            "Requirement already satisfied: autopage>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from cliff->optuna->-r requirements.txt (line 8)) (0.4.0)\n",
            "Requirement already satisfied: pbr!=2.1.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from cliff->optuna->-r requirements.txt (line 8)) (5.8.0)\n",
            "Requirement already satisfied: cmd2>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from cliff->optuna->-r requirements.txt (line 8)) (2.3.3)\n",
            "Requirement already satisfied: stevedore>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from cliff->optuna->-r requirements.txt (line 8)) (3.5.0)\n",
            "Requirement already satisfied: PrettyTable>=0.7.2 in /usr/local/lib/python3.7/dist-packages (from cliff->optuna->-r requirements.txt (line 8)) (3.0.0)\n",
            "Requirement already satisfied: pyperclip>=1.6 in /usr/local/lib/python3.7/dist-packages (from cmd2>=1.0.0->cliff->optuna->-r requirements.txt (line 8)) (1.8.2)\n",
            "Requirement already satisfied: wcwidth>=0.1.7 in /usr/local/lib/python3.7/dist-packages (from cmd2>=1.0.0->cliff->optuna->-r requirements.txt (line 8)) (0.2.5)\n",
            "Requirement already satisfied: attrs>=16.3.0 in /usr/local/lib/python3.7/dist-packages (from cmd2>=1.0.0->cliff->optuna->-r requirements.txt (line 8)) (21.4.0)\n",
            "Requirement already satisfied: MarkupSafe>=0.9.2 in /usr/local/lib/python3.7/dist-packages (from Mako->alembic->optuna->-r requirements.txt (line 8)) (2.0.1)\n",
            "Requirement already satisfied: py>=1.5.0 in /usr/local/lib/python3.7/dist-packages (from pytest->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (1.11.0)\n",
            "Requirement already satisfied: pluggy>=0.7 in /usr/local/lib/python3.7/dist-packages (from pytest->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (0.7.1)\n",
            "Requirement already satisfied: atomicwrites>=1.0 in /usr/local/lib/python3.7/dist-packages (from pytest->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (1.4.0)\n",
            "Requirement already satisfied: more-itertools>=4.0.0 in /usr/local/lib/python3.7/dist-packages (from pytest->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (8.12.0)\n",
            "Requirement already satisfied: coverage>=4.4 in /usr/local/lib/python3.7/dist-packages (from pytest-cov->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (6.2)\n",
            "Requirement already satisfied: execnet>=1.1 in /usr/local/lib/python3.7/dist-packages (from pytest-xdist->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (1.9.0)\n",
            "Requirement already satisfied: pytest-forked in /usr/local/lib/python3.7/dist-packages (from pytest-xdist->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (1.4.0)\n",
            "Requirement already satisfied: ninja>=1.10.0.post2 in /usr/local/lib/python3.7/dist-packages (from pytype->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (1.10.2.3)\n",
            "Requirement already satisfied: importlab>=0.7 in /usr/local/lib/python3.7/dist-packages (from pytype->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (0.7)\n",
            "Requirement already satisfied: libcst in /usr/local/lib/python3.7/dist-packages (from pytype->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (0.4.0)\n",
            "Requirement already satisfied: tabulate in /usr/local/lib/python3.7/dist-packages (from pytype->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (0.8.9)\n",
            "Requirement already satisfied: toml in /usr/local/lib/python3.7/dist-packages (from pytype->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (0.10.2)\n",
            "Requirement already satisfied: networkx>=2 in /usr/local/lib/python3.7/dist-packages (from importlab>=0.7->pytype->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (2.6.3)\n",
            "Requirement already satisfied: typing-inspect>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from libcst->pytype->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (0.7.1)\n",
            "Requirement already satisfied: Pygments>=2.0 in /usr/local/lib/python3.7/dist-packages (from sphinx->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (2.6.1)\n",
            "Requirement already satisfied: snowballstemmer>=1.1 in /usr/local/lib/python3.7/dist-packages (from sphinx->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (2.2.0)\n",
            "Requirement already satisfied: Jinja2>=2.3 in /usr/local/lib/python3.7/dist-packages (from sphinx->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (2.11.3)\n",
            "Requirement already satisfied: sphinxcontrib-devhelp in /usr/local/lib/python3.7/dist-packages (from sphinx->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (1.0.2)\n",
            "Requirement already satisfied: docutils<0.18,>=0.14 in /usr/local/lib/python3.7/dist-packages (from sphinx->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (0.17.1)\n",
            "Requirement already satisfied: sphinxcontrib-applehelp in /usr/local/lib/python3.7/dist-packages (from sphinx->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (1.0.2)\n",
            "Requirement already satisfied: alabaster<0.8,>=0.7 in /usr/local/lib/python3.7/dist-packages (from sphinx->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (0.7.12)\n",
            "Requirement already satisfied: babel>=1.3 in /usr/local/lib/python3.7/dist-packages (from sphinx->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (2.9.1)\n",
            "Requirement already satisfied: imagesize in /usr/local/lib/python3.7/dist-packages (from sphinx->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (1.3.0)\n",
            "Requirement already satisfied: sphinxcontrib-htmlhelp>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from sphinx->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (2.0.0)\n",
            "Requirement already satisfied: sphinxcontrib-qthelp in /usr/local/lib/python3.7/dist-packages (from sphinx->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (1.0.3)\n",
            "Requirement already satisfied: sphinxcontrib-jsmath in /usr/local/lib/python3.7/dist-packages (from sphinx->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (1.0.1)\n",
            "Requirement already satisfied: sphinxcontrib-serializinghtml>=1.1.5 in /usr/local/lib/python3.7/dist-packages (from sphinx->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (1.1.5)\n",
            "Requirement already satisfied: colorama in /usr/local/lib/python3.7/dist-packages (from sphinx-autobuild->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (0.4.4)\n",
            "Requirement already satisfied: livereload in /usr/local/lib/python3.7/dist-packages (from sphinx-autobuild->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (2.6.3)\n",
            "Requirement already satisfied: tornado in /usr/local/lib/python3.7/dist-packages (from livereload->sphinx-autobuild->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (5.1.1)\n",
            "Requirement already satisfied: PyEnchant>=3.1.1 in /usr/local/lib/python3.7/dist-packages (from sphinxcontrib.spelling->stable-baselines3[docs,extra,tests]>=1.3.1a8->-r requirements.txt (line 2)) (3.2.2)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6gJ-pAbF7zRZ"
      },
      "source": [
        "## Train an RL Agent\n",
        "\n",
        "\n",
        "The train agent can be found in the `logs/` folder.\n",
        "\n",
        "Here we will train A2C on CartPole-v1 environment for 100 000 steps. \n",
        "\n",
        "\n",
        "To train it on Pong (Atari), you just have to pass `--env PongNoFrameskip-v4`\n",
        "\n",
        "Note: You need to update `hyperparams/algo.yml` to support new environments. You can access it in the side panel of Google Colab. (see https://stackoverflow.com/questions/46986398/import-data-into-google-colaboratory)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9bIR_N7R11XI",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "02d5f220-b028-4e64-eec5-c189ebc4e64e"
      },
      "source": [
        "!python train.py --algo a2c --env CartPole-v1 --n-timesteps 10"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "========== CartPole-v1 ==========\n",
            "Seed: 3430206127\n",
            "Default hyperparameters for environment (ones being tuned will be overridden):\n",
            "OrderedDict([('ent_coef', 0.0),\n",
            "             ('n_envs', 8),\n",
            "             ('n_timesteps', 500000.0),\n",
            "             ('policy', 'MlpPolicy')])\n",
            "Using 8 environments\n",
            "Overwriting n_timesteps with n=10\n",
            "Creating test environment\n",
            "Using cpu device\n",
            "Log path: logs/a2c/CartPole-v1_1\n",
            "Saving to logs/a2c/CartPole-v1_1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-fHBq73665yD"
      },
      "source": [
        "#### Evaluate trained agent\n",
        "\n",
        "\n",
        "You can remove the `--folder logs/` to evaluate pretrained agent."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Bw8YuEgU6bT3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5bcf1dd0-1523-46a0-9920-8095fe7deebd"
      },
      "source": [
        "!python enjoy.py --algo a2c --env CartPole-v1 --no-render --n-timesteps 5000 --folder logs/"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Loading latest experiment, id=1\n",
            "Loading logs/a2c/CartPole-v1_1/CartPole-v1.zip\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 9.00\n",
            "Episode Length 9\n",
            "Episode Reward: 16.00\n",
            "Episode Length 16\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 9.00\n",
            "Episode Length 9\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 16.00\n",
            "Episode Length 16\n",
            "Episode Reward: 9.00\n",
            "Episode Length 9\n",
            "Episode Reward: 15.00\n",
            "Episode Length 15\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 15.00\n",
            "Episode Length 15\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 15.00\n",
            "Episode Length 15\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 32.00\n",
            "Episode Length 32\n",
            "Episode Reward: 31.00\n",
            "Episode Length 31\n",
            "Episode Reward: 31.00\n",
            "Episode Length 31\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 16.00\n",
            "Episode Length 16\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 32.00\n",
            "Episode Length 32\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 17.00\n",
            "Episode Length 17\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 15.00\n",
            "Episode Length 15\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 29.00\n",
            "Episode Length 29\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 16.00\n",
            "Episode Length 16\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 18.00\n",
            "Episode Length 18\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 32.00\n",
            "Episode Length 32\n",
            "Episode Reward: 15.00\n",
            "Episode Length 15\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 16.00\n",
            "Episode Length 16\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 9.00\n",
            "Episode Length 9\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 32.00\n",
            "Episode Length 32\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 16.00\n",
            "Episode Length 16\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 18.00\n",
            "Episode Length 18\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 9.00\n",
            "Episode Length 9\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 16.00\n",
            "Episode Length 16\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 16.00\n",
            "Episode Length 16\n",
            "Episode Reward: 16.00\n",
            "Episode Length 16\n",
            "Episode Reward: 16.00\n",
            "Episode Length 16\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 15.00\n",
            "Episode Length 15\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 34.00\n",
            "Episode Length 34\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 35.00\n",
            "Episode Length 35\n",
            "Episode Reward: 32.00\n",
            "Episode Length 32\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 16.00\n",
            "Episode Length 16\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 33.00\n",
            "Episode Length 33\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 18.00\n",
            "Episode Length 18\n",
            "Episode Reward: 35.00\n",
            "Episode Length 35\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 15.00\n",
            "Episode Length 15\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 15.00\n",
            "Episode Length 15\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 16.00\n",
            "Episode Length 16\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 15.00\n",
            "Episode Length 15\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 16.00\n",
            "Episode Length 16\n",
            "Episode Reward: 33.00\n",
            "Episode Length 33\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 16.00\n",
            "Episode Length 16\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 9.00\n",
            "Episode Length 9\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 16.00\n",
            "Episode Length 16\n",
            "Episode Reward: 33.00\n",
            "Episode Length 33\n",
            "Episode Reward: 29.00\n",
            "Episode Length 29\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 33.00\n",
            "Episode Length 33\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 21.00\n",
            "Episode Length 21\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 30.00\n",
            "Episode Length 30\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 15.00\n",
            "Episode Length 15\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 15.00\n",
            "Episode Length 15\n",
            "Episode Reward: 16.00\n",
            "Episode Length 16\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 9.00\n",
            "Episode Length 9\n",
            "Episode Reward: 15.00\n",
            "Episode Length 15\n",
            "Episode Reward: 16.00\n",
            "Episode Length 16\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 15.00\n",
            "Episode Length 15\n",
            "Episode Reward: 15.00\n",
            "Episode Length 15\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 17.00\n",
            "Episode Length 17\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 17.00\n",
            "Episode Length 17\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 16.00\n",
            "Episode Length 16\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 20.00\n",
            "Episode Length 20\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 36.00\n",
            "Episode Length 36\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 17.00\n",
            "Episode Length 17\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 37.00\n",
            "Episode Length 37\n",
            "Episode Reward: 9.00\n",
            "Episode Length 9\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 15.00\n",
            "Episode Length 15\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 17.00\n",
            "Episode Length 17\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 9.00\n",
            "Episode Length 9\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 9.00\n",
            "Episode Length 9\n",
            "Episode Reward: 17.00\n",
            "Episode Length 17\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 16.00\n",
            "Episode Length 16\n",
            "Episode Reward: 33.00\n",
            "Episode Length 33\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 15.00\n",
            "Episode Length 15\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 31.00\n",
            "Episode Length 31\n",
            "Episode Reward: 29.00\n",
            "Episode Length 29\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 16.00\n",
            "Episode Length 16\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 15.00\n",
            "Episode Length 15\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 35.00\n",
            "Episode Length 35\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 15.00\n",
            "Episode Length 15\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 15.00\n",
            "Episode Length 15\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 31.00\n",
            "Episode Length 31\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 30.00\n",
            "Episode Length 30\n",
            "Episode Reward: 33.00\n",
            "Episode Length 33\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 14.00\n",
            "Episode Length 14\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 10.00\n",
            "Episode Length 10\n",
            "Episode Reward: 15.00\n",
            "Episode Length 15\n",
            "Episode Reward: 16.00\n",
            "Episode Length 16\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 13.00\n",
            "Episode Length 13\n",
            "Episode Reward: 29.00\n",
            "Episode Length 29\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 20.00\n",
            "Episode Length 20\n",
            "Episode Reward: 11.00\n",
            "Episode Length 11\n",
            "Episode Reward: 12.00\n",
            "Episode Length 12\n",
            "Episode Reward: 35.00\n",
            "Episode Length 35\n",
            "357 Episodes\n",
            "Mean reward: 14.00 +/- 5.75\n",
            "Mean episode length: 14.00 +/- 5.75\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w5Il2J0VHPLC"
      },
      "source": [
        "#### Tune Hyperparameters\n",
        "\n",
        "We use [Optuna](https://optuna.org/) for optimizing the hyperparameters.\n",
        "\n",
        "Tune the hyperparameters for PPO, using a tpe sampler and median pruner, 2 parallels jobs,\n",
        "with a budget of 1000 trials and a maximum of 50000 steps"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "w2sC22eGHTH-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "eed29347-7446-45d7-9c68-5047e5cd12c7"
      },
      "source": [
        "!python train.py --algo ppo --env MountainCar-v0 -n 50 -optimize --n-trials 1 --n-jobs 2 --sampler tpe --pruner median"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "========== MountainCar-v0 ==========\n",
            "Seed: 2632319663\n",
            "Default hyperparameters for environment (ones being tuned will be overridden):\n",
            "OrderedDict([('ent_coef', 0.0),\n",
            "             ('gae_lambda', 0.98),\n",
            "             ('gamma', 0.99),\n",
            "             ('n_envs', 16),\n",
            "             ('n_epochs', 4),\n",
            "             ('n_steps', 16),\n",
            "             ('n_timesteps', 1000000.0),\n",
            "             ('normalize', True),\n",
            "             ('policy', 'MlpPolicy')])\n",
            "Using 16 environments\n",
            "Overwriting n_timesteps with n=50\n",
            "Normalization activated: {'gamma': 0.99}\n",
            "Optimizing hyperparameters\n",
            "Sampler: tpe - Pruner: median\n",
            "\u001b[32m[I 2022-01-18 00:46:41,638]\u001b[0m A new study created in memory with name: no-name-32e2ee8b-3225-45b6-82c8-6dcfcb061db1\u001b[0m\n",
            "/usr/local/lib/python3.7/dist-packages/optuna/study/study.py:397: FutureWarning:\n",
            "\n",
            "`n_jobs` argument has been deprecated in v2.7.0. This feature will be removed in v4.0.0. See https://github.com/optuna/optuna/releases/tag/v2.7.0.\n",
            "\n",
            "Normalization activated: {'gamma': 0.99}\n",
            "Normalization activated: {'gamma': 0.99, 'norm_reward': False}\n",
            "\u001b[32m[I 2022-01-18 00:46:48,479]\u001b[0m Trial 0 finished with value: -200.0 and parameters: {'batch_size': 256, 'n_steps': 8, 'gamma': 0.9999, 'learning_rate': 0.3270842923094914, 'ent_coef': 1.581420984681729e-07, 'clip_range': 0.4, 'n_epochs': 10, 'gae_lambda': 0.92, 'max_grad_norm': 5, 'vf_coef': 0.526214719917527, 'net_arch': 'small', 'activation_fn': 'relu'}. Best is trial 0 with value: -200.0.\u001b[0m\n",
            "Number of finished trials:  1\n",
            "Best trial:\n",
            "Value:  -200.0\n",
            "Params: \n",
            "    batch_size: 256\n",
            "    n_steps: 8\n",
            "    gamma: 0.9999\n",
            "    learning_rate: 0.3270842923094914\n",
            "    ent_coef: 1.581420984681729e-07\n",
            "    clip_range: 0.4\n",
            "    n_epochs: 10\n",
            "    gae_lambda: 0.92\n",
            "    max_grad_norm: 5\n",
            "    vf_coef: 0.526214719917527\n",
            "    net_arch: small\n",
            "    activation_fn: relu\n",
            "Writing report to logs/ppo/report_MountainCar-v0_1-trials-50-tpe-median_1642466808\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xVm9QPNVwKXN"
      },
      "source": [
        "### Record  a Video"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MPyfQxD5z26J"
      },
      "source": [
        "# Set up display; otherwise rendering will fail\n",
        "import os\n",
        "os.system(\"Xvfb :1 -screen 0 1024x768x24 &\")\n",
        "os.environ['DISPLAY'] = ':1'"
      ],
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ip3AauLzwNGP",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "734f1f85-609e-4c81-c798-da83b723403d"
      },
      "source": [
        "!python -m utils.record_video --algo a2c --env CartPole-v1 --exp-id 0 -f logs/ -n 1000"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Loading latest experiment, id=1\n",
            "Saving video to /content/rl-baselines3-zoo/logs/a2c/CartPole-v1_1/videos/final-model-a2c-CartPole-v1-step-0-to-step-1000.mp4\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qBuUfnzI8DN6"
      },
      "source": [
        "### Display the video"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZC3OTfpf8CXu"
      },
      "source": [
        "import base64\n",
        "from pathlib import Path\n",
        "\n",
        "from IPython import display as ipythondisplay\n",
        "\n",
        "def show_videos(video_path='', prefix=''):\n",
        "  \"\"\"\n",
        "  Taken from https://github.com/eleurent/highway-env\n",
        "\n",
        "  :param video_path: (str) Path to the folder containing videos\n",
        "  :param prefix: (str) Filter the video, showing only the only starting with this prefix\n",
        "  \"\"\"\n",
        "  html = []\n",
        "  for mp4 in Path(video_path).glob(\"{}*.mp4\".format(prefix)):\n",
        "      video_b64 = base64.b64encode(mp4.read_bytes())\n",
        "      html.append('''<video alt=\"{}\" autoplay \n",
        "                    loop controls style=\"height: 400px;\">\n",
        "                    <source src=\"data:video/mp4;base64,{}\" type=\"video/mp4\" />\n",
        "                </video>'''.format(mp4, video_b64.decode('ascii')))\n",
        "  ipythondisplay.display(ipythondisplay.HTML(data=\"<br>\".join(html)))"
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oKOjFuwK9HI0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "outputId": "530f2337-8998-4699-af45-b2a43d7da9f2"
      },
      "source": [
        "show_videos(video_path='logs/videos/', prefix='a2c')"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              ""
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RjdpP0HE8D2p"
      },
      "source": [
        "### Continue Training\n",
        "\n",
        "Here, we will continue training of the previous model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zgMZQJJF6u1C",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a0254ee7-16cb-4641-bf87-5af59aa1b88e"
      },
      "source": [
        "!python train.py --algo a2c --env CartPole-v1 --n-timesteps 50000 -i logs/a2c/CartPole-v1_1/CartPole-v1.zip"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Traceback (most recent call last):\n",
            "  File \"train.py\", line 15, in <module>\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GSaoyiAE8cVj"
      },
      "source": [
        "!python enjoy.py --algo a2c --env CartPole-v1 --no-render --n-timesteps 1000 --folder logs/"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jL9u4I1H-48O"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}