19/pretraining.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.10"
    },
    "colab": {
      "name": "pretraining.ipynb",
      "provenance": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/decoderkurt/HUF_RL_2022/blob/main/19/pretraining.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ShofuYy-8dLs"
      },
      "source": [
        "## Stable Baselines3 - Pretraining with Behavior Cloning\n",
        "\n",
        "\n",
        "Github repo: https://github.com/araffin/rl-tutorial-jnrr19\n",
        "\n",
        "Stable-Baselines3: https://github.com/DLR-RM/stable-baselines3\n",
        "\n",
        "Documentation: https://stable-baselines.readthedocs.io/en/master/\n",
        "\n",
        "RL Baselines3 zoo: https://github.com/DLR-RM/rl-baselines3-zoo\n",
        "\n",
        "\n",
        "[RL Baselines3 Zoo](https://github.com/DLR-RM/rl-baselines3-zoo) is a collection of pre-trained Reinforcement Learning agents using Stable-Baselines3.\n",
        "\n",
        "It also provides basic scripts for training, evaluating agents, tuning hyperparameters and recording videos.\n",
        "\n",
        "\n",
        "## Introduction\n",
        "\n",
        "In this notebook, you will learn how to record expert data, then pre-train an agent using this data\n",
        "and finally continue training with Stable-Baselines3.\n",
        "\n",
        "\n",
        "## Install Dependencies and Stable Baselines3 Using Pip\n",
        "\n",
        "List of full dependencies can be found in the [README](https://github.com/DLR-RM/stable-baselines3).\n",
        "\n",
        "Notebook originally created by [skervim](https://github.com/skervim)\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QwJ1LHlN9E33",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c726b6cf-d5b3-4de2-e9b2-7b50ce098ba6"
      },
      "source": [
        "# For Box2D env\n",
        "!apt-get install swig\n",
        "!pip install gym[box2d]\n",
        "!pip install stable-baselines3[extra]"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Reading package lists... Done\n",
            "Building dependency tree       \n",
            "Reading state information... Done\n",
            "The following additional packages will be installed:\n",
            "  swig3.0\n",
            "Suggested packages:\n",
            "  swig-doc swig-examples swig3.0-examples swig3.0-doc\n",
            "The following NEW packages will be installed:\n",
            "  swig swig3.0\n",
            "0 upgraded, 2 newly installed, 0 to remove and 37 not upgraded.\n",
            "Need to get 1,100 kB of archives.\n",
            "After this operation, 5,822 kB of additional disk space will be used.\n",
            "Get:1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 swig3.0 amd64 3.0.12-1 [1,094 kB]\n",
            "Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 swig amd64 3.0.12-1 [6,460 B]\n",
            "Fetched 1,100 kB in 3s (423 kB/s)\n",
            "Selecting previously unselected package swig3.0.\n",
            "(Reading database ... 155229 files and directories currently installed.)\n",
            "Preparing to unpack .../swig3.0_3.0.12-1_amd64.deb ...\n",
            "Unpacking swig3.0 (3.0.12-1) ...\n",
            "Selecting previously unselected package swig.\n",
            "Preparing to unpack .../swig_3.0.12-1_amd64.deb ...\n",
            "Unpacking swig (3.0.12-1) ...\n",
            "Setting up swig3.0 (3.0.12-1) ...\n",
            "Setting up swig (3.0.12-1) ...\n",
            "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...\n",
            "Requirement already satisfied: gym[box2d] in /usr/local/lib/python3.7/dist-packages (0.17.3)\n",
            "Requirement already satisfied: pyglet<=1.5.0,>=1.4.0 in /usr/local/lib/python3.7/dist-packages (from gym[box2d]) (1.5.0)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from gym[box2d]) (1.4.1)\n",
            "Requirement already satisfied: cloudpickle<1.7.0,>=1.2.0 in /usr/local/lib/python3.7/dist-packages (from gym[box2d]) (1.3.0)\n",
            "Requirement already satisfied: numpy>=1.10.4 in /usr/local/lib/python3.7/dist-packages (from gym[box2d]) (1.19.5)\n",
            "Collecting box2d-py~=2.3.5\n",
            "  Downloading box2d_py-2.3.8-cp37-cp37m-manylinux1_x86_64.whl (448 kB)\n",
            "\u001b[K     |&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;| 448 kB 4.3 MB/s \n",
            "\u001b[?25hRequirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from pyglet<=1.5.0,>=1.4.0->gym[box2d]) (0.16.0)\n",
            "Installing collected packages: box2d-py\n",
            "Successfully installed box2d-py-2.3.8\n",
            "Collecting stable-baselines3[extra]\n",
            "  Downloading stable_baselines3-1.3.0-py3-none-any.whl (174 kB)\n",
            "\u001b[K     |&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;&#9608;| 174 kB 4.3 MB/s \n",
            "\u001b[?25hRequirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[extra]) (3.2.2)\n",
            "Requirement already satisfied: cloudpickle in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[extra]) (1.3.0)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[extra]) (1.19.5)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[extra]) (1.1.5)\n",
            "Requirement already satisfied: torch>=1.8.1 in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[extra]) (1.10.0+cu111)\n",
            "Requirement already satisfied: gym<0.20,>=0.17 in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[extra]) (0.17.3)\n",
            "Requirement already satisfied: tensorboard>=2.2.0 in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[extra]) (2.7.0)\n",
            "Requirement already satisfied: pillow in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[extra]) (7.1.2)\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[extra]) (5.4.8)\n",
            "Requirement already satisfied: atari-py~=0.2.0 in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[extra]) (0.2.9)\n",
            "Requirement already satisfied: opencv-python in /usr/local/lib/python3.7/dist-packages (from stable-baselines3[extra]) (4.1.2.30)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from atari-py~=0.2.0->stable-baselines3[extra]) (1.15.0)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from gym<0.20,>=0.17->stable-baselines3[extra]) (1.4.1)\n",
            "Requirement already satisfied: pyglet<=1.5.0,>=1.4.0 in /usr/local/lib/python3.7/dist-packages (from gym<0.20,>=0.17->stable-baselines3[extra]) (1.5.0)\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from pyglet<=1.5.0,>=1.4.0->gym<0.20,>=0.17->stable-baselines3[extra]) (0.16.0)\n",
            "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[extra]) (0.37.1)\n",
            "Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[extra]) (1.35.0)\n",
            "Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[extra]) (0.12.0)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[extra]) (57.4.0)\n",
            "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[extra]) (1.8.1)\n",
            "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[extra]) (1.0.1)\n",
            "Requirement already satisfied: grpcio>=1.24.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[extra]) (1.43.0)\n",
            "Requirement already satisfied: protobuf>=3.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[extra]) (3.17.3)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[extra]) (3.3.6)\n",
            "Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[extra]) (2.23.0)\n",
            "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[extra]) (0.4.6)\n",
            "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard>=2.2.0->stable-baselines3[extra]) (0.6.1)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.2.0->stable-baselines3[extra]) (0.2.8)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.2.0->stable-baselines3[extra]) (4.8)\n",
            "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.2.0->stable-baselines3[extra]) (4.2.4)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=2.2.0->stable-baselines3[extra]) (1.3.0)\n",
            "Requirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard>=2.2.0->stable-baselines3[extra]) (4.10.0)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard>=2.2.0->stable-baselines3[extra]) (3.7.0)\n",
            "Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard>=2.2.0->stable-baselines3[extra]) (3.10.0.2)\n",
            "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard>=2.2.0->stable-baselines3[extra]) (0.4.8)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard>=2.2.0->stable-baselines3[extra]) (2.10)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard>=2.2.0->stable-baselines3[extra]) (1.24.3)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard>=2.2.0->stable-baselines3[extra]) (3.0.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard>=2.2.0->stable-baselines3[extra]) (2021.10.8)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard>=2.2.0->stable-baselines3[extra]) (3.1.1)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->stable-baselines3[extra]) (0.11.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->stable-baselines3[extra]) (1.3.2)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->stable-baselines3[extra]) (2.8.2)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->stable-baselines3[extra]) (3.0.6)\n",
            "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.7/dist-packages (from pandas->stable-baselines3[extra]) (2018.9)\n",
            "Installing collected packages: stable-baselines3\n",
            "Successfully installed stable-baselines3-1.3.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Tgx4AMZo8anP"
      },
      "source": [
        "import gym\n",
        "from tqdm import tqdm\n",
        "import numpy as np"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U_1ZNBum8ane"
      },
      "source": [
        "import torch as th\n",
        "import torch.nn as nn\n",
        "import torch.optim as optim\n",
        "from torch.optim.lr_scheduler import StepLR"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iaz2Szrl8anx"
      },
      "source": [
        "from stable_baselines3 import PPO, A2C, SAC, TD3\n",
        "from stable_baselines3.common.evaluation import evaluate_policy"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dS9lr70R9eU9"
      },
      "source": [
        "# Example for continuous actions\n",
        "# env_id = \"LunarLanderContinuous-v2\"\n",
        "\n",
        "# Example for discrete actions\n",
        "env_id = \"CartPole-v1\""
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eh88d4oR8an6"
      },
      "source": [
        "env = gym.make(env_id)"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rKCgHCc_8aoB"
      },
      "source": [
        "## Train Expert Model\n",
        "\n",
        "We create an expert RL agent and let it learn to solve a task by interacting with the evironment.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EkmIST0r8aoC",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "a7686dd4-96bc-4768-ce00-1da48ea34120"
      },
      "source": [
        "ppo_expert = PPO('MlpPolicy', env_id, verbose=1, create_eval_env=True)\n",
        "ppo_expert.learn(total_timesteps=3e4, eval_freq=10)\n",
        "ppo_expert.save(\"ppo_expert\")"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Using cpu device\n",
            "Creating environment from the given name 'CartPole-v1'\n",
            "Creating environment from the given name 'CartPole-v1'\n",
            "Wrapping the env with a `Monitor` wrapper\n",
            "Wrapping the env in a DummyVecEnv.\n",
            "Wrapping the env with a `Monitor` wrapper\n",
            "Wrapping the env in a DummyVecEnv.\n",
            "Eval num_timesteps=10, episode_reward=9.60 +/- 0.49\n",
            "Episode length: 9.60 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.6      |\n",
            "|    mean_reward     | 9.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 10       |\n",
            "---------------------------------\n",
            "New best mean reward!\n",
            "Eval num_timesteps=20, episode_reward=9.60 +/- 0.49\n",
            "Episode length: 9.60 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.6      |\n",
            "|    mean_reward     | 9.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 20       |\n",
            "---------------------------------\n",
            "Eval num_timesteps=30, episode_reward=8.20 +/- 0.40\n",
            "Episode length: 8.20 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.2      |\n",
            "|    mean_reward     | 8.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 30       |\n",
            "---------------------------------\n",
            "Eval num_timesteps=40, episode_reward=8.80 +/- 0.75\n",
            "Episode length: 8.80 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 40       |\n",
            "---------------------------------\n",
            "Eval num_timesteps=50, episode_reward=8.80 +/- 0.75\n",
            "Episode length: 8.80 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 50       |\n",
            "---------------------------------\n",
            "Eval num_timesteps=60, episode_reward=9.00 +/- 0.63\n",
            "Episode length: 9.00 +/- 0.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 60       |\n",
            "---------------------------------\n",
            "Eval num_timesteps=70, episode_reward=8.80 +/- 0.75\n",
            "Episode length: 8.80 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 70       |\n",
            "---------------------------------\n",
            "Eval num_timesteps=80, episode_reward=9.80 +/- 0.75\n",
            "Episode length: 9.80 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.8      |\n",
            "|    mean_reward     | 9.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 80       |\n",
            "---------------------------------\n",
            "New best mean reward!\n",
            "Eval num_timesteps=90, episode_reward=9.60 +/- 0.49\n",
            "Episode length: 9.60 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.6      |\n",
            "|    mean_reward     | 9.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 90       |\n",
            "---------------------------------\n",
            "Eval num_timesteps=100, episode_reward=9.00 +/- 0.63\n",
            "Episode length: 9.00 +/- 0.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 100      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=110, episode_reward=9.00 +/- 0.89\n",
            "Episode length: 9.00 +/- 0.89\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 110      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=120, episode_reward=9.00 +/- 0.63\n",
            "Episode length: 9.00 +/- 0.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 120      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=130, episode_reward=8.60 +/- 0.49\n",
            "Episode length: 8.60 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.6      |\n",
            "|    mean_reward     | 8.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 130      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=140, episode_reward=9.60 +/- 0.80\n",
            "Episode length: 9.60 +/- 0.80\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.6      |\n",
            "|    mean_reward     | 9.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 140      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=150, episode_reward=9.00 +/- 0.63\n",
            "Episode length: 9.00 +/- 0.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 150      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=160, episode_reward=8.80 +/- 0.75\n",
            "Episode length: 8.80 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 160      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=170, episode_reward=9.40 +/- 0.49\n",
            "Episode length: 9.40 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 170      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=180, episode_reward=9.40 +/- 1.02\n",
            "Episode length: 9.40 +/- 1.02\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 180      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=190, episode_reward=8.80 +/- 0.40\n",
            "Episode length: 8.80 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 190      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=200, episode_reward=9.00 +/- 0.89\n",
            "Episode length: 9.00 +/- 0.89\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 200      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=210, episode_reward=9.40 +/- 1.02\n",
            "Episode length: 9.40 +/- 1.02\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 210      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=220, episode_reward=8.80 +/- 0.75\n",
            "Episode length: 8.80 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 220      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=230, episode_reward=9.40 +/- 0.49\n",
            "Episode length: 9.40 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 230      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=240, episode_reward=10.00 +/- 0.63\n",
            "Episode length: 10.00 +/- 0.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 10       |\n",
            "|    mean_reward     | 10       |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 240      |\n",
            "---------------------------------\n",
            "New best mean reward!\n",
            "Eval num_timesteps=250, episode_reward=9.20 +/- 0.98\n",
            "Episode length: 9.20 +/- 0.98\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 250      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=260, episode_reward=9.20 +/- 0.40\n",
            "Episode length: 9.20 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 260      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=270, episode_reward=9.80 +/- 0.40\n",
            "Episode length: 9.80 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.8      |\n",
            "|    mean_reward     | 9.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 270      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=280, episode_reward=9.00 +/- 0.00\n",
            "Episode length: 9.00 +/- 0.00\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 280      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=290, episode_reward=9.00 +/- 0.63\n",
            "Episode length: 9.00 +/- 0.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 290      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=300, episode_reward=8.80 +/- 0.98\n",
            "Episode length: 8.80 +/- 0.98\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 300      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=310, episode_reward=8.60 +/- 0.80\n",
            "Episode length: 8.60 +/- 0.80\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.6      |\n",
            "|    mean_reward     | 8.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 310      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=320, episode_reward=9.20 +/- 0.75\n",
            "Episode length: 9.20 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 320      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=330, episode_reward=9.00 +/- 0.63\n",
            "Episode length: 9.00 +/- 0.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 330      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=340, episode_reward=9.00 +/- 0.89\n",
            "Episode length: 9.00 +/- 0.89\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 340      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=350, episode_reward=8.80 +/- 0.75\n",
            "Episode length: 8.80 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 350      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=360, episode_reward=8.80 +/- 0.75\n",
            "Episode length: 8.80 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 360      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=370, episode_reward=9.40 +/- 0.80\n",
            "Episode length: 9.40 +/- 0.80\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 370      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=380, episode_reward=8.60 +/- 0.49\n",
            "Episode length: 8.60 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.6      |\n",
            "|    mean_reward     | 8.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 380      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=390, episode_reward=9.20 +/- 0.75\n",
            "Episode length: 9.20 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 390      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=400, episode_reward=8.80 +/- 0.40\n",
            "Episode length: 8.80 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 400      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=410, episode_reward=8.80 +/- 0.75\n",
            "Episode length: 8.80 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 410      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=420, episode_reward=9.40 +/- 0.49\n",
            "Episode length: 9.40 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 420      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=430, episode_reward=9.00 +/- 0.00\n",
            "Episode length: 9.00 +/- 0.00\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 430      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=440, episode_reward=9.20 +/- 0.40\n",
            "Episode length: 9.20 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 440      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=450, episode_reward=9.20 +/- 0.98\n",
            "Episode length: 9.20 +/- 0.98\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 450      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=460, episode_reward=9.40 +/- 0.80\n",
            "Episode length: 9.40 +/- 0.80\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 460      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=470, episode_reward=9.40 +/- 0.49\n",
            "Episode length: 9.40 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 470      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=480, episode_reward=9.40 +/- 0.80\n",
            "Episode length: 9.40 +/- 0.80\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 480      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=490, episode_reward=9.20 +/- 0.75\n",
            "Episode length: 9.20 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 490      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=500, episode_reward=8.60 +/- 0.49\n",
            "Episode length: 8.60 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.6      |\n",
            "|    mean_reward     | 8.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 500      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=510, episode_reward=9.00 +/- 1.10\n",
            "Episode length: 9.00 +/- 1.10\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 510      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=520, episode_reward=9.60 +/- 0.49\n",
            "Episode length: 9.60 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.6      |\n",
            "|    mean_reward     | 9.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 520      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=530, episode_reward=10.00 +/- 0.00\n",
            "Episode length: 10.00 +/- 0.00\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 10       |\n",
            "|    mean_reward     | 10       |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 530      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=540, episode_reward=8.60 +/- 0.49\n",
            "Episode length: 8.60 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.6      |\n",
            "|    mean_reward     | 8.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 540      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=550, episode_reward=9.40 +/- 0.49\n",
            "Episode length: 9.40 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 550      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=560, episode_reward=9.00 +/- 0.89\n",
            "Episode length: 9.00 +/- 0.89\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 560      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=570, episode_reward=9.40 +/- 0.49\n",
            "Episode length: 9.40 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 570      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=580, episode_reward=9.60 +/- 0.49\n",
            "Episode length: 9.60 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.6      |\n",
            "|    mean_reward     | 9.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 580      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=590, episode_reward=9.00 +/- 0.00\n",
            "Episode length: 9.00 +/- 0.00\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 590      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=600, episode_reward=9.20 +/- 0.40\n",
            "Episode length: 9.20 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 600      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=610, episode_reward=9.00 +/- 0.00\n",
            "Episode length: 9.00 +/- 0.00\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 610      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=620, episode_reward=9.20 +/- 0.98\n",
            "Episode length: 9.20 +/- 0.98\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 620      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=630, episode_reward=9.60 +/- 0.49\n",
            "Episode length: 9.60 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.6      |\n",
            "|    mean_reward     | 9.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 630      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=640, episode_reward=9.40 +/- 0.49\n",
            "Episode length: 9.40 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 640      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=650, episode_reward=9.60 +/- 0.80\n",
            "Episode length: 9.60 +/- 0.80\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.6      |\n",
            "|    mean_reward     | 9.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 650      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=660, episode_reward=9.00 +/- 0.63\n",
            "Episode length: 9.00 +/- 0.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 660      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=670, episode_reward=9.40 +/- 0.80\n",
            "Episode length: 9.40 +/- 0.80\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 670      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=680, episode_reward=9.40 +/- 0.49\n",
            "Episode length: 9.40 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 680      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=690, episode_reward=9.60 +/- 0.49\n",
            "Episode length: 9.60 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.6      |\n",
            "|    mean_reward     | 9.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 690      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=700, episode_reward=9.40 +/- 1.02\n",
            "Episode length: 9.40 +/- 1.02\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 700      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=710, episode_reward=8.80 +/- 0.75\n",
            "Episode length: 8.80 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 710      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=720, episode_reward=8.60 +/- 0.49\n",
            "Episode length: 8.60 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.6      |\n",
            "|    mean_reward     | 8.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 720      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=730, episode_reward=9.40 +/- 0.80\n",
            "Episode length: 9.40 +/- 0.80\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 730      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=740, episode_reward=8.80 +/- 0.40\n",
            "Episode length: 8.80 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 740      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=750, episode_reward=9.00 +/- 0.89\n",
            "Episode length: 9.00 +/- 0.89\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 750      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=760, episode_reward=9.00 +/- 0.63\n",
            "Episode length: 9.00 +/- 0.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 760      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=770, episode_reward=8.80 +/- 0.75\n",
            "Episode length: 8.80 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 770      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=780, episode_reward=9.20 +/- 0.40\n",
            "Episode length: 9.20 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 780      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=790, episode_reward=9.40 +/- 1.20\n",
            "Episode length: 9.40 +/- 1.20\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 790      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=800, episode_reward=9.20 +/- 0.40\n",
            "Episode length: 9.20 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 800      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=810, episode_reward=9.20 +/- 0.40\n",
            "Episode length: 9.20 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 810      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=820, episode_reward=9.20 +/- 0.75\n",
            "Episode length: 9.20 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 820      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=830, episode_reward=8.80 +/- 0.75\n",
            "Episode length: 8.80 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 830      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=840, episode_reward=9.00 +/- 0.89\n",
            "Episode length: 9.00 +/- 0.89\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 840      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=850, episode_reward=9.60 +/- 0.80\n",
            "Episode length: 9.60 +/- 0.80\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.6      |\n",
            "|    mean_reward     | 9.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 850      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=860, episode_reward=9.80 +/- 0.40\n",
            "Episode length: 9.80 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.8      |\n",
            "|    mean_reward     | 9.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 860      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=870, episode_reward=9.00 +/- 0.00\n",
            "Episode length: 9.00 +/- 0.00\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 870      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=880, episode_reward=8.40 +/- 0.49\n",
            "Episode length: 8.40 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.4      |\n",
            "|    mean_reward     | 8.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 880      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=890, episode_reward=9.20 +/- 0.75\n",
            "Episode length: 9.20 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 890      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=900, episode_reward=9.00 +/- 0.63\n",
            "Episode length: 9.00 +/- 0.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 900      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=910, episode_reward=8.80 +/- 0.40\n",
            "Episode length: 8.80 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 910      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=920, episode_reward=9.20 +/- 0.75\n",
            "Episode length: 9.20 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 920      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=930, episode_reward=9.20 +/- 0.75\n",
            "Episode length: 9.20 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 930      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=940, episode_reward=9.00 +/- 0.63\n",
            "Episode length: 9.00 +/- 0.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 940      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=950, episode_reward=8.40 +/- 0.80\n",
            "Episode length: 8.40 +/- 0.80\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.4      |\n",
            "|    mean_reward     | 8.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 950      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=960, episode_reward=9.60 +/- 0.49\n",
            "Episode length: 9.60 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.6      |\n",
            "|    mean_reward     | 9.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 960      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=970, episode_reward=8.80 +/- 0.40\n",
            "Episode length: 8.80 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 970      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=980, episode_reward=8.80 +/- 0.40\n",
            "Episode length: 8.80 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 980      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=990, episode_reward=9.40 +/- 0.49\n",
            "Episode length: 9.40 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 990      |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1000, episode_reward=8.80 +/- 0.75\n",
            "Episode length: 8.80 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1000     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1010, episode_reward=8.40 +/- 0.49\n",
            "Episode length: 8.40 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.4      |\n",
            "|    mean_reward     | 8.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1010     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1020, episode_reward=9.40 +/- 0.49\n",
            "Episode length: 9.40 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1020     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1030, episode_reward=8.60 +/- 0.49\n",
            "Episode length: 8.60 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.6      |\n",
            "|    mean_reward     | 8.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1030     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1040, episode_reward=9.40 +/- 0.49\n",
            "Episode length: 9.40 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1040     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1050, episode_reward=9.40 +/- 0.49\n",
            "Episode length: 9.40 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1050     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1060, episode_reward=9.40 +/- 0.49\n",
            "Episode length: 9.40 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1060     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1070, episode_reward=8.60 +/- 0.80\n",
            "Episode length: 8.60 +/- 0.80\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.6      |\n",
            "|    mean_reward     | 8.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1070     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1080, episode_reward=9.60 +/- 0.49\n",
            "Episode length: 9.60 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.6      |\n",
            "|    mean_reward     | 9.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1080     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1090, episode_reward=9.00 +/- 0.63\n",
            "Episode length: 9.00 +/- 0.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1090     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1100, episode_reward=8.40 +/- 0.49\n",
            "Episode length: 8.40 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.4      |\n",
            "|    mean_reward     | 8.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1100     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1110, episode_reward=9.00 +/- 0.63\n",
            "Episode length: 9.00 +/- 0.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1110     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1120, episode_reward=9.00 +/- 0.00\n",
            "Episode length: 9.00 +/- 0.00\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1120     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1130, episode_reward=9.20 +/- 0.40\n",
            "Episode length: 9.20 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1130     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1140, episode_reward=9.60 +/- 0.49\n",
            "Episode length: 9.60 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.6      |\n",
            "|    mean_reward     | 9.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1140     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1150, episode_reward=9.00 +/- 0.00\n",
            "Episode length: 9.00 +/- 0.00\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1150     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1160, episode_reward=8.80 +/- 0.40\n",
            "Episode length: 8.80 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1160     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1170, episode_reward=8.40 +/- 0.49\n",
            "Episode length: 8.40 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.4      |\n",
            "|    mean_reward     | 8.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1170     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1180, episode_reward=9.60 +/- 0.49\n",
            "Episode length: 9.60 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.6      |\n",
            "|    mean_reward     | 9.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1180     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1190, episode_reward=8.80 +/- 0.75\n",
            "Episode length: 8.80 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1190     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1200, episode_reward=9.00 +/- 0.63\n",
            "Episode length: 9.00 +/- 0.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1200     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1210, episode_reward=9.20 +/- 0.40\n",
            "Episode length: 9.20 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1210     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1220, episode_reward=9.60 +/- 0.80\n",
            "Episode length: 9.60 +/- 0.80\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.6      |\n",
            "|    mean_reward     | 9.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1220     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1230, episode_reward=9.00 +/- 0.63\n",
            "Episode length: 9.00 +/- 0.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1230     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1240, episode_reward=9.80 +/- 0.40\n",
            "Episode length: 9.80 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.8      |\n",
            "|    mean_reward     | 9.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1240     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1250, episode_reward=9.40 +/- 0.49\n",
            "Episode length: 9.40 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1250     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1260, episode_reward=9.00 +/- 0.89\n",
            "Episode length: 9.00 +/- 0.89\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1260     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1270, episode_reward=9.40 +/- 0.49\n",
            "Episode length: 9.40 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1270     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1280, episode_reward=8.60 +/- 0.49\n",
            "Episode length: 8.60 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.6      |\n",
            "|    mean_reward     | 8.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1280     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1290, episode_reward=9.60 +/- 0.80\n",
            "Episode length: 9.60 +/- 0.80\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.6      |\n",
            "|    mean_reward     | 9.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1290     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1300, episode_reward=9.20 +/- 0.75\n",
            "Episode length: 9.20 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1300     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1310, episode_reward=9.20 +/- 0.40\n",
            "Episode length: 9.20 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1310     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1320, episode_reward=9.20 +/- 0.75\n",
            "Episode length: 9.20 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1320     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1330, episode_reward=9.00 +/- 0.63\n",
            "Episode length: 9.00 +/- 0.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1330     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1340, episode_reward=9.40 +/- 0.80\n",
            "Episode length: 9.40 +/- 0.80\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1340     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1350, episode_reward=9.40 +/- 0.49\n",
            "Episode length: 9.40 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1350     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1360, episode_reward=8.80 +/- 0.75\n",
            "Episode length: 8.80 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1360     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1370, episode_reward=9.20 +/- 0.40\n",
            "Episode length: 9.20 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1370     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1380, episode_reward=9.00 +/- 0.00\n",
            "Episode length: 9.00 +/- 0.00\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1380     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1390, episode_reward=9.00 +/- 0.00\n",
            "Episode length: 9.00 +/- 0.00\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1390     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1400, episode_reward=9.60 +/- 0.49\n",
            "Episode length: 9.60 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.6      |\n",
            "|    mean_reward     | 9.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1400     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1410, episode_reward=9.40 +/- 0.49\n",
            "Episode length: 9.40 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1410     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1420, episode_reward=9.20 +/- 0.75\n",
            "Episode length: 9.20 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1420     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1430, episode_reward=9.20 +/- 0.40\n",
            "Episode length: 9.20 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1430     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1440, episode_reward=8.80 +/- 0.40\n",
            "Episode length: 8.80 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1440     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1450, episode_reward=9.00 +/- 0.63\n",
            "Episode length: 9.00 +/- 0.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1450     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1460, episode_reward=8.80 +/- 0.40\n",
            "Episode length: 8.80 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1460     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1470, episode_reward=9.00 +/- 0.00\n",
            "Episode length: 9.00 +/- 0.00\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1470     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1480, episode_reward=9.40 +/- 0.49\n",
            "Episode length: 9.40 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1480     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1490, episode_reward=9.40 +/- 0.49\n",
            "Episode length: 9.40 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1490     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1500, episode_reward=9.20 +/- 0.75\n",
            "Episode length: 9.20 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1500     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1510, episode_reward=9.20 +/- 0.75\n",
            "Episode length: 9.20 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1510     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1520, episode_reward=8.80 +/- 0.75\n",
            "Episode length: 8.80 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1520     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1530, episode_reward=8.80 +/- 0.75\n",
            "Episode length: 8.80 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1530     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1540, episode_reward=9.00 +/- 0.63\n",
            "Episode length: 9.00 +/- 0.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1540     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1550, episode_reward=9.00 +/- 0.63\n",
            "Episode length: 9.00 +/- 0.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1550     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1560, episode_reward=8.80 +/- 0.75\n",
            "Episode length: 8.80 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1560     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1570, episode_reward=9.40 +/- 0.80\n",
            "Episode length: 9.40 +/- 0.80\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1570     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1580, episode_reward=9.20 +/- 0.40\n",
            "Episode length: 9.20 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1580     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1590, episode_reward=8.60 +/- 0.80\n",
            "Episode length: 8.60 +/- 0.80\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.6      |\n",
            "|    mean_reward     | 8.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1590     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1600, episode_reward=9.40 +/- 1.02\n",
            "Episode length: 9.40 +/- 1.02\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1600     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1610, episode_reward=9.40 +/- 0.80\n",
            "Episode length: 9.40 +/- 0.80\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1610     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1620, episode_reward=8.80 +/- 0.40\n",
            "Episode length: 8.80 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1620     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1630, episode_reward=9.00 +/- 0.63\n",
            "Episode length: 9.00 +/- 0.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1630     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1640, episode_reward=8.60 +/- 0.49\n",
            "Episode length: 8.60 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.6      |\n",
            "|    mean_reward     | 8.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1640     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1650, episode_reward=9.00 +/- 0.63\n",
            "Episode length: 9.00 +/- 0.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1650     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1660, episode_reward=8.80 +/- 0.40\n",
            "Episode length: 8.80 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1660     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1670, episode_reward=9.20 +/- 0.75\n",
            "Episode length: 9.20 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1670     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1680, episode_reward=9.60 +/- 0.49\n",
            "Episode length: 9.60 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.6      |\n",
            "|    mean_reward     | 9.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1680     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1690, episode_reward=9.40 +/- 0.49\n",
            "Episode length: 9.40 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1690     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1700, episode_reward=8.80 +/- 0.75\n",
            "Episode length: 8.80 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1700     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1710, episode_reward=9.00 +/- 0.63\n",
            "Episode length: 9.00 +/- 0.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1710     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1720, episode_reward=9.00 +/- 0.63\n",
            "Episode length: 9.00 +/- 0.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1720     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1730, episode_reward=9.20 +/- 0.75\n",
            "Episode length: 9.20 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1730     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1740, episode_reward=9.00 +/- 0.63\n",
            "Episode length: 9.00 +/- 0.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1740     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1750, episode_reward=9.20 +/- 0.98\n",
            "Episode length: 9.20 +/- 0.98\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1750     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1760, episode_reward=8.80 +/- 0.40\n",
            "Episode length: 8.80 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1760     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1770, episode_reward=9.20 +/- 0.40\n",
            "Episode length: 9.20 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1770     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1780, episode_reward=9.20 +/- 0.40\n",
            "Episode length: 9.20 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1780     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1790, episode_reward=8.80 +/- 0.75\n",
            "Episode length: 8.80 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1790     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1800, episode_reward=8.80 +/- 0.40\n",
            "Episode length: 8.80 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1800     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1810, episode_reward=9.00 +/- 0.63\n",
            "Episode length: 9.00 +/- 0.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1810     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1820, episode_reward=9.00 +/- 0.89\n",
            "Episode length: 9.00 +/- 0.89\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1820     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1830, episode_reward=9.00 +/- 0.63\n",
            "Episode length: 9.00 +/- 0.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1830     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1840, episode_reward=9.00 +/- 0.63\n",
            "Episode length: 9.00 +/- 0.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1840     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1850, episode_reward=9.80 +/- 0.40\n",
            "Episode length: 9.80 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.8      |\n",
            "|    mean_reward     | 9.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1850     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1860, episode_reward=9.00 +/- 0.89\n",
            "Episode length: 9.00 +/- 0.89\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1860     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1870, episode_reward=9.20 +/- 0.75\n",
            "Episode length: 9.20 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1870     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1880, episode_reward=9.00 +/- 0.89\n",
            "Episode length: 9.00 +/- 0.89\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1880     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1890, episode_reward=8.80 +/- 0.40\n",
            "Episode length: 8.80 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.8      |\n",
            "|    mean_reward     | 8.8      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1890     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1900, episode_reward=8.60 +/- 0.49\n",
            "Episode length: 8.60 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.6      |\n",
            "|    mean_reward     | 8.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1900     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1910, episode_reward=9.00 +/- 0.63\n",
            "Episode length: 9.00 +/- 0.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1910     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1920, episode_reward=9.20 +/- 0.75\n",
            "Episode length: 9.20 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1920     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1930, episode_reward=9.00 +/- 0.63\n",
            "Episode length: 9.00 +/- 0.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1930     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1940, episode_reward=8.20 +/- 0.40\n",
            "Episode length: 8.20 +/- 0.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.2      |\n",
            "|    mean_reward     | 8.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1940     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1950, episode_reward=9.00 +/- 0.63\n",
            "Episode length: 9.00 +/- 0.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1950     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1960, episode_reward=9.60 +/- 0.49\n",
            "Episode length: 9.60 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.6      |\n",
            "|    mean_reward     | 9.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1960     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1970, episode_reward=9.00 +/- 0.63\n",
            "Episode length: 9.00 +/- 0.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1970     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1980, episode_reward=9.00 +/- 0.00\n",
            "Episode length: 9.00 +/- 0.00\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1980     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=1990, episode_reward=8.60 +/- 0.80\n",
            "Episode length: 8.60 +/- 0.80\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.6      |\n",
            "|    mean_reward     | 8.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 1990     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2000, episode_reward=9.40 +/- 0.49\n",
            "Episode length: 9.40 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2000     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2010, episode_reward=8.60 +/- 0.49\n",
            "Episode length: 8.60 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 8.6      |\n",
            "|    mean_reward     | 8.6      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2010     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2020, episode_reward=9.20 +/- 0.75\n",
            "Episode length: 9.20 +/- 0.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.2      |\n",
            "|    mean_reward     | 9.2      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2020     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2030, episode_reward=9.00 +/- 0.89\n",
            "Episode length: 9.00 +/- 0.89\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9        |\n",
            "|    mean_reward     | 9        |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2030     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2040, episode_reward=9.40 +/- 0.49\n",
            "Episode length: 9.40 +/- 0.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 9.4      |\n",
            "|    mean_reward     | 9.4      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2040     |\n",
            "---------------------------------\n",
            "---------------------------------\n",
            "| rollout/           |          |\n",
            "|    ep_len_mean     | 22.3     |\n",
            "|    ep_rew_mean     | 22.3     |\n",
            "| time/              |          |\n",
            "|    fps             | 247      |\n",
            "|    iterations      | 1        |\n",
            "|    time_elapsed    | 8        |\n",
            "|    total_timesteps | 2048     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2050, episode_reward=306.40 +/- 48.04\n",
            "Episode length: 306.40 +/- 48.04\n",
            "-----------------------------------------\n",
            "| eval/                   |             |\n",
            "|    mean_ep_length       | 306         |\n",
            "|    mean_reward          | 306         |\n",
            "| time/                   |             |\n",
            "|    total_timesteps      | 2050        |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.008525699 |\n",
            "|    clip_fraction        | 0.0874      |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -0.687      |\n",
            "|    explained_variance   | -0.0129     |\n",
            "|    learning_rate        | 0.0003      |\n",
            "|    loss                 | 8.14        |\n",
            "|    n_updates            | 10          |\n",
            "|    policy_gradient_loss | -0.0125     |\n",
            "|    value_loss           | 55.6        |\n",
            "-----------------------------------------\n",
            "New best mean reward!\n",
            "Eval num_timesteps=2060, episode_reward=284.00 +/- 115.47\n",
            "Episode length: 284.00 +/- 115.47\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 284      |\n",
            "|    mean_reward     | 284      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2060     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2070, episode_reward=245.00 +/- 127.79\n",
            "Episode length: 245.00 +/- 127.79\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 245      |\n",
            "|    mean_reward     | 245      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2070     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2080, episode_reward=244.80 +/- 51.53\n",
            "Episode length: 244.80 +/- 51.53\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 245      |\n",
            "|    mean_reward     | 245      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2080     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2090, episode_reward=282.60 +/- 82.49\n",
            "Episode length: 282.60 +/- 82.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 283      |\n",
            "|    mean_reward     | 283      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2090     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2100, episode_reward=309.40 +/- 100.22\n",
            "Episode length: 309.40 +/- 100.22\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 309      |\n",
            "|    mean_reward     | 309      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2100     |\n",
            "---------------------------------\n",
            "New best mean reward!\n",
            "Eval num_timesteps=2110, episode_reward=319.40 +/- 42.61\n",
            "Episode length: 319.40 +/- 42.61\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 319      |\n",
            "|    mean_reward     | 319      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2110     |\n",
            "---------------------------------\n",
            "New best mean reward!\n",
            "Eval num_timesteps=2120, episode_reward=336.20 +/- 95.36\n",
            "Episode length: 336.20 +/- 95.36\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 336      |\n",
            "|    mean_reward     | 336      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2120     |\n",
            "---------------------------------\n",
            "New best mean reward!\n",
            "Eval num_timesteps=2130, episode_reward=411.00 +/- 31.05\n",
            "Episode length: 411.00 +/- 31.05\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 411      |\n",
            "|    mean_reward     | 411      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2130     |\n",
            "---------------------------------\n",
            "New best mean reward!\n",
            "Eval num_timesteps=2140, episode_reward=252.80 +/- 89.90\n",
            "Episode length: 252.80 +/- 89.90\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 253      |\n",
            "|    mean_reward     | 253      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2140     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2150, episode_reward=333.40 +/- 51.65\n",
            "Episode length: 333.40 +/- 51.65\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 333      |\n",
            "|    mean_reward     | 333      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2150     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2160, episode_reward=235.40 +/- 58.23\n",
            "Episode length: 235.40 +/- 58.23\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 235      |\n",
            "|    mean_reward     | 235      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2160     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2170, episode_reward=325.80 +/- 119.76\n",
            "Episode length: 325.80 +/- 119.76\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 326      |\n",
            "|    mean_reward     | 326      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2170     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2180, episode_reward=335.40 +/- 136.52\n",
            "Episode length: 335.40 +/- 136.52\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 335      |\n",
            "|    mean_reward     | 335      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2180     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2190, episode_reward=333.20 +/- 77.31\n",
            "Episode length: 333.20 +/- 77.31\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 333      |\n",
            "|    mean_reward     | 333      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2190     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2200, episode_reward=276.40 +/- 114.63\n",
            "Episode length: 276.40 +/- 114.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 276      |\n",
            "|    mean_reward     | 276      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2200     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2210, episode_reward=236.80 +/- 77.31\n",
            "Episode length: 236.80 +/- 77.31\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 237      |\n",
            "|    mean_reward     | 237      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2210     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2220, episode_reward=209.80 +/- 50.70\n",
            "Episode length: 209.80 +/- 50.70\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 210      |\n",
            "|    mean_reward     | 210      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2220     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2230, episode_reward=276.40 +/- 92.31\n",
            "Episode length: 276.40 +/- 92.31\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 276      |\n",
            "|    mean_reward     | 276      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2230     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2240, episode_reward=327.20 +/- 128.79\n",
            "Episode length: 327.20 +/- 128.79\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 327      |\n",
            "|    mean_reward     | 327      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2240     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2250, episode_reward=266.20 +/- 62.91\n",
            "Episode length: 266.20 +/- 62.91\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 266      |\n",
            "|    mean_reward     | 266      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2250     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2260, episode_reward=283.40 +/- 65.04\n",
            "Episode length: 283.40 +/- 65.04\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 283      |\n",
            "|    mean_reward     | 283      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2260     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2270, episode_reward=294.20 +/- 144.48\n",
            "Episode length: 294.20 +/- 144.48\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 294      |\n",
            "|    mean_reward     | 294      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2270     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2280, episode_reward=334.20 +/- 87.99\n",
            "Episode length: 334.20 +/- 87.99\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 334      |\n",
            "|    mean_reward     | 334      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2280     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2290, episode_reward=302.60 +/- 74.78\n",
            "Episode length: 302.60 +/- 74.78\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 303      |\n",
            "|    mean_reward     | 303      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2290     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2300, episode_reward=315.60 +/- 98.35\n",
            "Episode length: 315.60 +/- 98.35\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 316      |\n",
            "|    mean_reward     | 316      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2300     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2310, episode_reward=322.00 +/- 109.55\n",
            "Episode length: 322.00 +/- 109.55\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 322      |\n",
            "|    mean_reward     | 322      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2310     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2320, episode_reward=321.60 +/- 112.48\n",
            "Episode length: 321.60 +/- 112.48\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 322      |\n",
            "|    mean_reward     | 322      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2320     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2330, episode_reward=356.00 +/- 79.39\n",
            "Episode length: 356.00 +/- 79.39\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 356      |\n",
            "|    mean_reward     | 356      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2330     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2340, episode_reward=313.20 +/- 96.56\n",
            "Episode length: 313.20 +/- 96.56\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 313      |\n",
            "|    mean_reward     | 313      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2340     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2350, episode_reward=258.00 +/- 124.24\n",
            "Episode length: 258.00 +/- 124.24\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 258      |\n",
            "|    mean_reward     | 258      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2350     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2360, episode_reward=256.20 +/- 82.97\n",
            "Episode length: 256.20 +/- 82.97\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 256      |\n",
            "|    mean_reward     | 256      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2360     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2370, episode_reward=336.00 +/- 123.82\n",
            "Episode length: 336.00 +/- 123.82\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 336      |\n",
            "|    mean_reward     | 336      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2370     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2380, episode_reward=303.00 +/- 87.37\n",
            "Episode length: 303.00 +/- 87.37\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 303      |\n",
            "|    mean_reward     | 303      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2380     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2390, episode_reward=407.40 +/- 86.41\n",
            "Episode length: 407.40 +/- 86.41\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 407      |\n",
            "|    mean_reward     | 407      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2390     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2400, episode_reward=209.20 +/- 39.18\n",
            "Episode length: 209.20 +/- 39.18\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 209      |\n",
            "|    mean_reward     | 209      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2400     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2410, episode_reward=278.40 +/- 40.37\n",
            "Episode length: 278.40 +/- 40.37\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 278      |\n",
            "|    mean_reward     | 278      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2410     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2420, episode_reward=305.20 +/- 105.38\n",
            "Episode length: 305.20 +/- 105.38\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 305      |\n",
            "|    mean_reward     | 305      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2420     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2430, episode_reward=237.20 +/- 40.86\n",
            "Episode length: 237.20 +/- 40.86\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 237      |\n",
            "|    mean_reward     | 237      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2430     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2440, episode_reward=363.40 +/- 112.20\n",
            "Episode length: 363.40 +/- 112.20\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 363      |\n",
            "|    mean_reward     | 363      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2440     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2450, episode_reward=286.80 +/- 104.01\n",
            "Episode length: 286.80 +/- 104.01\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 287      |\n",
            "|    mean_reward     | 287      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2450     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2460, episode_reward=315.60 +/- 58.85\n",
            "Episode length: 315.60 +/- 58.85\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 316      |\n",
            "|    mean_reward     | 316      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2460     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2470, episode_reward=335.60 +/- 102.26\n",
            "Episode length: 335.60 +/- 102.26\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 336      |\n",
            "|    mean_reward     | 336      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2470     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2480, episode_reward=331.80 +/- 136.27\n",
            "Episode length: 331.80 +/- 136.27\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 332      |\n",
            "|    mean_reward     | 332      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2480     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2490, episode_reward=278.00 +/- 75.43\n",
            "Episode length: 278.00 +/- 75.43\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 278      |\n",
            "|    mean_reward     | 278      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2490     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2500, episode_reward=284.60 +/- 113.27\n",
            "Episode length: 284.60 +/- 113.27\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 285      |\n",
            "|    mean_reward     | 285      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2500     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2510, episode_reward=225.40 +/- 38.42\n",
            "Episode length: 225.40 +/- 38.42\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 225      |\n",
            "|    mean_reward     | 225      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2510     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2520, episode_reward=287.20 +/- 86.35\n",
            "Episode length: 287.20 +/- 86.35\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 287      |\n",
            "|    mean_reward     | 287      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2520     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2530, episode_reward=241.80 +/- 45.21\n",
            "Episode length: 241.80 +/- 45.21\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 242      |\n",
            "|    mean_reward     | 242      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2530     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2540, episode_reward=221.60 +/- 45.57\n",
            "Episode length: 221.60 +/- 45.57\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 222      |\n",
            "|    mean_reward     | 222      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2540     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2550, episode_reward=346.80 +/- 66.81\n",
            "Episode length: 346.80 +/- 66.81\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 347      |\n",
            "|    mean_reward     | 347      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2550     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2560, episode_reward=231.00 +/- 78.53\n",
            "Episode length: 231.00 +/- 78.53\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 231      |\n",
            "|    mean_reward     | 231      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2560     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2570, episode_reward=258.60 +/- 67.97\n",
            "Episode length: 258.60 +/- 67.97\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 259      |\n",
            "|    mean_reward     | 259      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2570     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2580, episode_reward=254.00 +/- 51.98\n",
            "Episode length: 254.00 +/- 51.98\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 254      |\n",
            "|    mean_reward     | 254      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2580     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2590, episode_reward=249.20 +/- 30.95\n",
            "Episode length: 249.20 +/- 30.95\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 249      |\n",
            "|    mean_reward     | 249      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2590     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2600, episode_reward=301.40 +/- 56.96\n",
            "Episode length: 301.40 +/- 56.96\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 301      |\n",
            "|    mean_reward     | 301      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2600     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2610, episode_reward=323.00 +/- 139.07\n",
            "Episode length: 323.00 +/- 139.07\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 323      |\n",
            "|    mean_reward     | 323      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2610     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2620, episode_reward=236.00 +/- 51.37\n",
            "Episode length: 236.00 +/- 51.37\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 236      |\n",
            "|    mean_reward     | 236      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2620     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2630, episode_reward=313.60 +/- 117.30\n",
            "Episode length: 313.60 +/- 117.30\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 314      |\n",
            "|    mean_reward     | 314      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2630     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2640, episode_reward=279.20 +/- 64.70\n",
            "Episode length: 279.20 +/- 64.70\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 279      |\n",
            "|    mean_reward     | 279      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2640     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2650, episode_reward=234.20 +/- 87.07\n",
            "Episode length: 234.20 +/- 87.07\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 234      |\n",
            "|    mean_reward     | 234      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2650     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2660, episode_reward=293.80 +/- 96.70\n",
            "Episode length: 293.80 +/- 96.70\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 294      |\n",
            "|    mean_reward     | 294      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2660     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2670, episode_reward=298.20 +/- 116.38\n",
            "Episode length: 298.20 +/- 116.38\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 298      |\n",
            "|    mean_reward     | 298      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2670     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2680, episode_reward=296.80 +/- 106.30\n",
            "Episode length: 296.80 +/- 106.30\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 297      |\n",
            "|    mean_reward     | 297      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2680     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2690, episode_reward=225.60 +/- 53.79\n",
            "Episode length: 225.60 +/- 53.79\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 226      |\n",
            "|    mean_reward     | 226      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2690     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2700, episode_reward=301.20 +/- 114.75\n",
            "Episode length: 301.20 +/- 114.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 301      |\n",
            "|    mean_reward     | 301      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2700     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2710, episode_reward=203.80 +/- 66.05\n",
            "Episode length: 203.80 +/- 66.05\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 204      |\n",
            "|    mean_reward     | 204      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2710     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2720, episode_reward=230.60 +/- 75.56\n",
            "Episode length: 230.60 +/- 75.56\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 231      |\n",
            "|    mean_reward     | 231      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2720     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2730, episode_reward=219.00 +/- 59.91\n",
            "Episode length: 219.00 +/- 59.91\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 219      |\n",
            "|    mean_reward     | 219      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2730     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2740, episode_reward=218.80 +/- 38.79\n",
            "Episode length: 218.80 +/- 38.79\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 219      |\n",
            "|    mean_reward     | 219      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2740     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2750, episode_reward=227.80 +/- 60.60\n",
            "Episode length: 227.80 +/- 60.60\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 228      |\n",
            "|    mean_reward     | 228      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2750     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2760, episode_reward=254.60 +/- 125.64\n",
            "Episode length: 254.60 +/- 125.64\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 255      |\n",
            "|    mean_reward     | 255      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2760     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2770, episode_reward=256.40 +/- 86.34\n",
            "Episode length: 256.40 +/- 86.34\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 256      |\n",
            "|    mean_reward     | 256      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2770     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2780, episode_reward=296.20 +/- 65.82\n",
            "Episode length: 296.20 +/- 65.82\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 296      |\n",
            "|    mean_reward     | 296      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2780     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2790, episode_reward=350.20 +/- 67.37\n",
            "Episode length: 350.20 +/- 67.37\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 350      |\n",
            "|    mean_reward     | 350      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2790     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2800, episode_reward=249.20 +/- 73.81\n",
            "Episode length: 249.20 +/- 73.81\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 249      |\n",
            "|    mean_reward     | 249      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2800     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2810, episode_reward=416.80 +/- 82.29\n",
            "Episode length: 416.80 +/- 82.29\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 417      |\n",
            "|    mean_reward     | 417      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2810     |\n",
            "---------------------------------\n",
            "New best mean reward!\n",
            "Eval num_timesteps=2820, episode_reward=326.40 +/- 86.70\n",
            "Episode length: 326.40 +/- 86.70\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 326      |\n",
            "|    mean_reward     | 326      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2820     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2830, episode_reward=288.20 +/- 115.77\n",
            "Episode length: 288.20 +/- 115.77\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 288      |\n",
            "|    mean_reward     | 288      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2830     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2840, episode_reward=300.20 +/- 102.73\n",
            "Episode length: 300.20 +/- 102.73\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 300      |\n",
            "|    mean_reward     | 300      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2840     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2850, episode_reward=319.00 +/- 93.53\n",
            "Episode length: 319.00 +/- 93.53\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 319      |\n",
            "|    mean_reward     | 319      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2850     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2860, episode_reward=221.80 +/- 26.19\n",
            "Episode length: 221.80 +/- 26.19\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 222      |\n",
            "|    mean_reward     | 222      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2860     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2870, episode_reward=341.80 +/- 115.06\n",
            "Episode length: 341.80 +/- 115.06\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 342      |\n",
            "|    mean_reward     | 342      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2870     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2880, episode_reward=191.80 +/- 50.33\n",
            "Episode length: 191.80 +/- 50.33\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 192      |\n",
            "|    mean_reward     | 192      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2880     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2890, episode_reward=357.20 +/- 84.64\n",
            "Episode length: 357.20 +/- 84.64\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 357      |\n",
            "|    mean_reward     | 357      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2890     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2900, episode_reward=233.60 +/- 65.55\n",
            "Episode length: 233.60 +/- 65.55\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 234      |\n",
            "|    mean_reward     | 234      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2900     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2910, episode_reward=222.20 +/- 76.26\n",
            "Episode length: 222.20 +/- 76.26\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 222      |\n",
            "|    mean_reward     | 222      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2910     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2920, episode_reward=276.40 +/- 130.00\n",
            "Episode length: 276.40 +/- 130.00\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 276      |\n",
            "|    mean_reward     | 276      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2920     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2930, episode_reward=196.60 +/- 35.10\n",
            "Episode length: 196.60 +/- 35.10\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 197      |\n",
            "|    mean_reward     | 197      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2930     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2940, episode_reward=221.00 +/- 54.04\n",
            "Episode length: 221.00 +/- 54.04\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 221      |\n",
            "|    mean_reward     | 221      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2940     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2950, episode_reward=268.80 +/- 100.89\n",
            "Episode length: 268.80 +/- 100.89\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 269      |\n",
            "|    mean_reward     | 269      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2950     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2960, episode_reward=290.60 +/- 87.19\n",
            "Episode length: 290.60 +/- 87.19\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 291      |\n",
            "|    mean_reward     | 291      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2960     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2970, episode_reward=274.60 +/- 72.46\n",
            "Episode length: 274.60 +/- 72.46\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 275      |\n",
            "|    mean_reward     | 275      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2970     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2980, episode_reward=276.20 +/- 87.46\n",
            "Episode length: 276.20 +/- 87.46\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 276      |\n",
            "|    mean_reward     | 276      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2980     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=2990, episode_reward=268.00 +/- 98.77\n",
            "Episode length: 268.00 +/- 98.77\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 268      |\n",
            "|    mean_reward     | 268      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 2990     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3000, episode_reward=263.00 +/- 82.66\n",
            "Episode length: 263.00 +/- 82.66\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 263      |\n",
            "|    mean_reward     | 263      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3000     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3010, episode_reward=203.40 +/- 54.46\n",
            "Episode length: 203.40 +/- 54.46\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 203      |\n",
            "|    mean_reward     | 203      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3010     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3020, episode_reward=328.00 +/- 121.50\n",
            "Episode length: 328.00 +/- 121.50\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 328      |\n",
            "|    mean_reward     | 328      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3020     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3030, episode_reward=350.00 +/- 129.90\n",
            "Episode length: 350.00 +/- 129.90\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 350      |\n",
            "|    mean_reward     | 350      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3030     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3040, episode_reward=258.00 +/- 39.06\n",
            "Episode length: 258.00 +/- 39.06\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 258      |\n",
            "|    mean_reward     | 258      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3040     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3050, episode_reward=262.80 +/- 57.91\n",
            "Episode length: 262.80 +/- 57.91\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 263      |\n",
            "|    mean_reward     | 263      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3050     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3060, episode_reward=307.00 +/- 104.82\n",
            "Episode length: 307.00 +/- 104.82\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 307      |\n",
            "|    mean_reward     | 307      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3060     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3070, episode_reward=210.40 +/- 70.30\n",
            "Episode length: 210.40 +/- 70.30\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 210      |\n",
            "|    mean_reward     | 210      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3070     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3080, episode_reward=256.40 +/- 77.57\n",
            "Episode length: 256.40 +/- 77.57\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 256      |\n",
            "|    mean_reward     | 256      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3080     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3090, episode_reward=260.00 +/- 102.49\n",
            "Episode length: 260.00 +/- 102.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 260      |\n",
            "|    mean_reward     | 260      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3090     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3100, episode_reward=194.80 +/- 25.29\n",
            "Episode length: 194.80 +/- 25.29\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 195      |\n",
            "|    mean_reward     | 195      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3100     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3110, episode_reward=236.20 +/- 87.71\n",
            "Episode length: 236.20 +/- 87.71\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 236      |\n",
            "|    mean_reward     | 236      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3110     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3120, episode_reward=272.20 +/- 98.11\n",
            "Episode length: 272.20 +/- 98.11\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 272      |\n",
            "|    mean_reward     | 272      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3120     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3130, episode_reward=215.00 +/- 30.98\n",
            "Episode length: 215.00 +/- 30.98\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 215      |\n",
            "|    mean_reward     | 215      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3130     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3140, episode_reward=302.40 +/- 81.62\n",
            "Episode length: 302.40 +/- 81.62\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 302      |\n",
            "|    mean_reward     | 302      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3140     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3150, episode_reward=254.80 +/- 127.01\n",
            "Episode length: 254.80 +/- 127.01\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 255      |\n",
            "|    mean_reward     | 255      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3150     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3160, episode_reward=294.20 +/- 94.75\n",
            "Episode length: 294.20 +/- 94.75\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 294      |\n",
            "|    mean_reward     | 294      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3160     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3170, episode_reward=325.60 +/- 142.91\n",
            "Episode length: 325.60 +/- 142.91\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 326      |\n",
            "|    mean_reward     | 326      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3170     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3180, episode_reward=273.80 +/- 115.39\n",
            "Episode length: 273.80 +/- 115.39\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 274      |\n",
            "|    mean_reward     | 274      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3180     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3190, episode_reward=322.80 +/- 121.63\n",
            "Episode length: 322.80 +/- 121.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 323      |\n",
            "|    mean_reward     | 323      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3190     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3200, episode_reward=290.20 +/- 118.60\n",
            "Episode length: 290.20 +/- 118.60\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 290      |\n",
            "|    mean_reward     | 290      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3200     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3210, episode_reward=315.80 +/- 103.96\n",
            "Episode length: 315.80 +/- 103.96\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 316      |\n",
            "|    mean_reward     | 316      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3210     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3220, episode_reward=256.80 +/- 70.51\n",
            "Episode length: 256.80 +/- 70.51\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 257      |\n",
            "|    mean_reward     | 257      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3220     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3230, episode_reward=313.00 +/- 88.29\n",
            "Episode length: 313.00 +/- 88.29\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 313      |\n",
            "|    mean_reward     | 313      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3230     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3240, episode_reward=270.00 +/- 80.48\n",
            "Episode length: 270.00 +/- 80.48\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 270      |\n",
            "|    mean_reward     | 270      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3240     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3250, episode_reward=255.80 +/- 45.82\n",
            "Episode length: 255.80 +/- 45.82\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 256      |\n",
            "|    mean_reward     | 256      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3250     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3260, episode_reward=345.80 +/- 115.06\n",
            "Episode length: 345.80 +/- 115.06\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 346      |\n",
            "|    mean_reward     | 346      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3260     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3270, episode_reward=242.40 +/- 111.02\n",
            "Episode length: 242.40 +/- 111.02\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 242      |\n",
            "|    mean_reward     | 242      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3270     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3280, episode_reward=243.40 +/- 85.18\n",
            "Episode length: 243.40 +/- 85.18\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 243      |\n",
            "|    mean_reward     | 243      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3280     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3290, episode_reward=223.60 +/- 41.96\n",
            "Episode length: 223.60 +/- 41.96\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 224      |\n",
            "|    mean_reward     | 224      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3290     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3300, episode_reward=251.80 +/- 49.71\n",
            "Episode length: 251.80 +/- 49.71\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 252      |\n",
            "|    mean_reward     | 252      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3300     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3310, episode_reward=334.40 +/- 103.27\n",
            "Episode length: 334.40 +/- 103.27\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 334      |\n",
            "|    mean_reward     | 334      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3310     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3320, episode_reward=248.20 +/- 72.22\n",
            "Episode length: 248.20 +/- 72.22\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 248      |\n",
            "|    mean_reward     | 248      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3320     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3330, episode_reward=353.80 +/- 128.25\n",
            "Episode length: 353.80 +/- 128.25\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 354      |\n",
            "|    mean_reward     | 354      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3330     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3340, episode_reward=324.40 +/- 90.05\n",
            "Episode length: 324.40 +/- 90.05\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 324      |\n",
            "|    mean_reward     | 324      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3340     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3350, episode_reward=248.80 +/- 88.69\n",
            "Episode length: 248.80 +/- 88.69\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 249      |\n",
            "|    mean_reward     | 249      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3350     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3360, episode_reward=420.20 +/- 62.49\n",
            "Episode length: 420.20 +/- 62.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 420      |\n",
            "|    mean_reward     | 420      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3360     |\n",
            "---------------------------------\n",
            "New best mean reward!\n",
            "Eval num_timesteps=3370, episode_reward=254.40 +/- 46.93\n",
            "Episode length: 254.40 +/- 46.93\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 254      |\n",
            "|    mean_reward     | 254      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3370     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3380, episode_reward=222.60 +/- 68.48\n",
            "Episode length: 222.60 +/- 68.48\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 223      |\n",
            "|    mean_reward     | 223      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3380     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3390, episode_reward=263.60 +/- 69.19\n",
            "Episode length: 263.60 +/- 69.19\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 264      |\n",
            "|    mean_reward     | 264      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3390     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3400, episode_reward=287.20 +/- 114.72\n",
            "Episode length: 287.20 +/- 114.72\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 287      |\n",
            "|    mean_reward     | 287      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3400     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3410, episode_reward=239.80 +/- 55.01\n",
            "Episode length: 239.80 +/- 55.01\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 240      |\n",
            "|    mean_reward     | 240      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3410     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3420, episode_reward=272.60 +/- 116.92\n",
            "Episode length: 272.60 +/- 116.92\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 273      |\n",
            "|    mean_reward     | 273      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3420     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3430, episode_reward=321.20 +/- 74.26\n",
            "Episode length: 321.20 +/- 74.26\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 321      |\n",
            "|    mean_reward     | 321      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3430     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3440, episode_reward=273.20 +/- 117.56\n",
            "Episode length: 273.20 +/- 117.56\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 273      |\n",
            "|    mean_reward     | 273      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3440     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3450, episode_reward=278.20 +/- 116.40\n",
            "Episode length: 278.20 +/- 116.40\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 278      |\n",
            "|    mean_reward     | 278      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3450     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3460, episode_reward=320.00 +/- 129.70\n",
            "Episode length: 320.00 +/- 129.70\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 320      |\n",
            "|    mean_reward     | 320      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3460     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3470, episode_reward=265.40 +/- 89.87\n",
            "Episode length: 265.40 +/- 89.87\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 265      |\n",
            "|    mean_reward     | 265      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3470     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3480, episode_reward=319.60 +/- 81.67\n",
            "Episode length: 319.60 +/- 81.67\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 320      |\n",
            "|    mean_reward     | 320      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3480     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3490, episode_reward=263.40 +/- 48.95\n",
            "Episode length: 263.40 +/- 48.95\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 263      |\n",
            "|    mean_reward     | 263      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3490     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3500, episode_reward=347.40 +/- 127.80\n",
            "Episode length: 347.40 +/- 127.80\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 347      |\n",
            "|    mean_reward     | 347      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3500     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3510, episode_reward=243.40 +/- 114.33\n",
            "Episode length: 243.40 +/- 114.33\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 243      |\n",
            "|    mean_reward     | 243      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3510     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3520, episode_reward=335.60 +/- 82.80\n",
            "Episode length: 335.60 +/- 82.80\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 336      |\n",
            "|    mean_reward     | 336      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3520     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3530, episode_reward=205.00 +/- 55.32\n",
            "Episode length: 205.00 +/- 55.32\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 205      |\n",
            "|    mean_reward     | 205      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3530     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3540, episode_reward=263.20 +/- 73.93\n",
            "Episode length: 263.20 +/- 73.93\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 263      |\n",
            "|    mean_reward     | 263      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3540     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3550, episode_reward=286.40 +/- 65.87\n",
            "Episode length: 286.40 +/- 65.87\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 286      |\n",
            "|    mean_reward     | 286      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3550     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3560, episode_reward=262.60 +/- 102.80\n",
            "Episode length: 262.60 +/- 102.80\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 263      |\n",
            "|    mean_reward     | 263      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3560     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3570, episode_reward=373.40 +/- 127.88\n",
            "Episode length: 373.40 +/- 127.88\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 373      |\n",
            "|    mean_reward     | 373      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3570     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3580, episode_reward=265.20 +/- 56.47\n",
            "Episode length: 265.20 +/- 56.47\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 265      |\n",
            "|    mean_reward     | 265      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3580     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3590, episode_reward=277.80 +/- 115.71\n",
            "Episode length: 277.80 +/- 115.71\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 278      |\n",
            "|    mean_reward     | 278      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3590     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3600, episode_reward=276.80 +/- 107.25\n",
            "Episode length: 276.80 +/- 107.25\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 277      |\n",
            "|    mean_reward     | 277      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3600     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3610, episode_reward=296.80 +/- 86.07\n",
            "Episode length: 296.80 +/- 86.07\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 297      |\n",
            "|    mean_reward     | 297      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3610     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3620, episode_reward=243.40 +/- 78.54\n",
            "Episode length: 243.40 +/- 78.54\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 243      |\n",
            "|    mean_reward     | 243      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3620     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3630, episode_reward=219.20 +/- 58.61\n",
            "Episode length: 219.20 +/- 58.61\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 219      |\n",
            "|    mean_reward     | 219      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3630     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3640, episode_reward=251.20 +/- 57.79\n",
            "Episode length: 251.20 +/- 57.79\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 251      |\n",
            "|    mean_reward     | 251      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3640     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3650, episode_reward=239.20 +/- 62.44\n",
            "Episode length: 239.20 +/- 62.44\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 239      |\n",
            "|    mean_reward     | 239      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3650     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3660, episode_reward=346.20 +/- 95.56\n",
            "Episode length: 346.20 +/- 95.56\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 346      |\n",
            "|    mean_reward     | 346      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3660     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3670, episode_reward=244.00 +/- 81.07\n",
            "Episode length: 244.00 +/- 81.07\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 244      |\n",
            "|    mean_reward     | 244      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3670     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3680, episode_reward=287.80 +/- 105.05\n",
            "Episode length: 287.80 +/- 105.05\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 288      |\n",
            "|    mean_reward     | 288      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3680     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3690, episode_reward=234.80 +/- 104.15\n",
            "Episode length: 234.80 +/- 104.15\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 235      |\n",
            "|    mean_reward     | 235      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3690     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3700, episode_reward=274.80 +/- 75.23\n",
            "Episode length: 274.80 +/- 75.23\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 275      |\n",
            "|    mean_reward     | 275      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3700     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3710, episode_reward=380.80 +/- 102.09\n",
            "Episode length: 380.80 +/- 102.09\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 381      |\n",
            "|    mean_reward     | 381      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3710     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3720, episode_reward=232.80 +/- 86.53\n",
            "Episode length: 232.80 +/- 86.53\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 233      |\n",
            "|    mean_reward     | 233      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3720     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3730, episode_reward=356.00 +/- 74.16\n",
            "Episode length: 356.00 +/- 74.16\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 356      |\n",
            "|    mean_reward     | 356      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3730     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3740, episode_reward=271.80 +/- 98.65\n",
            "Episode length: 271.80 +/- 98.65\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 272      |\n",
            "|    mean_reward     | 272      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3740     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3750, episode_reward=314.20 +/- 102.72\n",
            "Episode length: 314.20 +/- 102.72\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 314      |\n",
            "|    mean_reward     | 314      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3750     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3760, episode_reward=289.60 +/- 84.83\n",
            "Episode length: 289.60 +/- 84.83\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 290      |\n",
            "|    mean_reward     | 290      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3760     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3770, episode_reward=330.00 +/- 91.55\n",
            "Episode length: 330.00 +/- 91.55\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 330      |\n",
            "|    mean_reward     | 330      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3770     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3780, episode_reward=265.00 +/- 81.43\n",
            "Episode length: 265.00 +/- 81.43\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 265      |\n",
            "|    mean_reward     | 265      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3780     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3790, episode_reward=298.40 +/- 50.51\n",
            "Episode length: 298.40 +/- 50.51\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 298      |\n",
            "|    mean_reward     | 298      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3790     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3800, episode_reward=312.20 +/- 117.13\n",
            "Episode length: 312.20 +/- 117.13\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 312      |\n",
            "|    mean_reward     | 312      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3800     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3810, episode_reward=322.60 +/- 93.44\n",
            "Episode length: 322.60 +/- 93.44\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 323      |\n",
            "|    mean_reward     | 323      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3810     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3820, episode_reward=340.80 +/- 110.69\n",
            "Episode length: 340.80 +/- 110.69\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 341      |\n",
            "|    mean_reward     | 341      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3820     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3830, episode_reward=266.00 +/- 67.21\n",
            "Episode length: 266.00 +/- 67.21\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 266      |\n",
            "|    mean_reward     | 266      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3830     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3840, episode_reward=277.60 +/- 52.28\n",
            "Episode length: 277.60 +/- 52.28\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 278      |\n",
            "|    mean_reward     | 278      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3840     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3850, episode_reward=257.20 +/- 56.94\n",
            "Episode length: 257.20 +/- 56.94\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 257      |\n",
            "|    mean_reward     | 257      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3850     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3860, episode_reward=236.60 +/- 65.33\n",
            "Episode length: 236.60 +/- 65.33\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 237      |\n",
            "|    mean_reward     | 237      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3860     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3870, episode_reward=206.00 +/- 46.29\n",
            "Episode length: 206.00 +/- 46.29\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 206      |\n",
            "|    mean_reward     | 206      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3870     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3880, episode_reward=247.40 +/- 60.91\n",
            "Episode length: 247.40 +/- 60.91\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 247      |\n",
            "|    mean_reward     | 247      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3880     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3890, episode_reward=300.40 +/- 52.56\n",
            "Episode length: 300.40 +/- 52.56\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 300      |\n",
            "|    mean_reward     | 300      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3890     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3900, episode_reward=252.40 +/- 73.06\n",
            "Episode length: 252.40 +/- 73.06\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 252      |\n",
            "|    mean_reward     | 252      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3900     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3910, episode_reward=214.20 +/- 43.37\n",
            "Episode length: 214.20 +/- 43.37\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 214      |\n",
            "|    mean_reward     | 214      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3910     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3920, episode_reward=211.40 +/- 40.95\n",
            "Episode length: 211.40 +/- 40.95\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 211      |\n",
            "|    mean_reward     | 211      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3920     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3930, episode_reward=315.40 +/- 64.08\n",
            "Episode length: 315.40 +/- 64.08\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 315      |\n",
            "|    mean_reward     | 315      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3930     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3940, episode_reward=343.20 +/- 116.26\n",
            "Episode length: 343.20 +/- 116.26\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 343      |\n",
            "|    mean_reward     | 343      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3940     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3950, episode_reward=262.60 +/- 77.49\n",
            "Episode length: 262.60 +/- 77.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 263      |\n",
            "|    mean_reward     | 263      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3950     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3960, episode_reward=246.20 +/- 74.11\n",
            "Episode length: 246.20 +/- 74.11\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 246      |\n",
            "|    mean_reward     | 246      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3960     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3970, episode_reward=201.60 +/- 33.72\n",
            "Episode length: 201.60 +/- 33.72\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 202      |\n",
            "|    mean_reward     | 202      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3970     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3980, episode_reward=287.80 +/- 64.81\n",
            "Episode length: 287.80 +/- 64.81\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 288      |\n",
            "|    mean_reward     | 288      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3980     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=3990, episode_reward=313.00 +/- 114.46\n",
            "Episode length: 313.00 +/- 114.46\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 313      |\n",
            "|    mean_reward     | 313      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 3990     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4000, episode_reward=244.80 +/- 94.09\n",
            "Episode length: 244.80 +/- 94.09\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 245      |\n",
            "|    mean_reward     | 245      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4000     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4010, episode_reward=287.60 +/- 128.80\n",
            "Episode length: 287.60 +/- 128.80\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 288      |\n",
            "|    mean_reward     | 288      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4010     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4020, episode_reward=292.00 +/- 70.74\n",
            "Episode length: 292.00 +/- 70.74\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 292      |\n",
            "|    mean_reward     | 292      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4020     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4030, episode_reward=249.60 +/- 64.72\n",
            "Episode length: 249.60 +/- 64.72\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 250      |\n",
            "|    mean_reward     | 250      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4030     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4040, episode_reward=346.60 +/- 108.00\n",
            "Episode length: 346.60 +/- 108.00\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 347      |\n",
            "|    mean_reward     | 347      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4040     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4050, episode_reward=282.40 +/- 113.26\n",
            "Episode length: 282.40 +/- 113.26\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 282      |\n",
            "|    mean_reward     | 282      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4050     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4060, episode_reward=245.20 +/- 62.76\n",
            "Episode length: 245.20 +/- 62.76\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 245      |\n",
            "|    mean_reward     | 245      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4060     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4070, episode_reward=295.80 +/- 82.84\n",
            "Episode length: 295.80 +/- 82.84\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 296      |\n",
            "|    mean_reward     | 296      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4070     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4080, episode_reward=277.80 +/- 88.06\n",
            "Episode length: 277.80 +/- 88.06\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 278      |\n",
            "|    mean_reward     | 278      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4080     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4090, episode_reward=280.80 +/- 110.14\n",
            "Episode length: 280.80 +/- 110.14\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 281      |\n",
            "|    mean_reward     | 281      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4090     |\n",
            "---------------------------------\n",
            "---------------------------------\n",
            "| rollout/           |          |\n",
            "|    ep_len_mean     | 28.5     |\n",
            "|    ep_rew_mean     | 28.5     |\n",
            "| time/              |          |\n",
            "|    fps             | 23       |\n",
            "|    iterations      | 2        |\n",
            "|    time_elapsed    | 175      |\n",
            "|    total_timesteps | 4096     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4100, episode_reward=97.20 +/- 9.70\n",
            "Episode length: 97.20 +/- 9.70\n",
            "------------------------------------------\n",
            "| eval/                   |              |\n",
            "|    mean_ep_length       | 97.2         |\n",
            "|    mean_reward          | 97.2         |\n",
            "| time/                   |              |\n",
            "|    total_timesteps      | 4100         |\n",
            "| train/                  |              |\n",
            "|    approx_kl            | 0.0096143745 |\n",
            "|    clip_fraction        | 0.0653       |\n",
            "|    clip_range           | 0.2          |\n",
            "|    entropy_loss         | -0.668       |\n",
            "|    explained_variance   | 0.0648       |\n",
            "|    learning_rate        | 0.0003       |\n",
            "|    loss                 | 14.8         |\n",
            "|    n_updates            | 20           |\n",
            "|    policy_gradient_loss | -0.0173      |\n",
            "|    value_loss           | 42.9         |\n",
            "------------------------------------------\n",
            "Eval num_timesteps=4110, episode_reward=89.40 +/- 16.89\n",
            "Episode length: 89.40 +/- 16.89\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 89.4     |\n",
            "|    mean_reward     | 89.4     |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4110     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4120, episode_reward=172.20 +/- 74.51\n",
            "Episode length: 172.20 +/- 74.51\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 172      |\n",
            "|    mean_reward     | 172      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4120     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4130, episode_reward=100.80 +/- 6.05\n",
            "Episode length: 100.80 +/- 6.05\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 101      |\n",
            "|    mean_reward     | 101      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4130     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4140, episode_reward=132.80 +/- 28.49\n",
            "Episode length: 132.80 +/- 28.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 133      |\n",
            "|    mean_reward     | 133      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4140     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4150, episode_reward=115.80 +/- 40.74\n",
            "Episode length: 115.80 +/- 40.74\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 116      |\n",
            "|    mean_reward     | 116      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4150     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4160, episode_reward=160.60 +/- 169.78\n",
            "Episode length: 160.60 +/- 169.78\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 161      |\n",
            "|    mean_reward     | 161      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4160     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4170, episode_reward=93.60 +/- 22.71\n",
            "Episode length: 93.60 +/- 22.71\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 93.6     |\n",
            "|    mean_reward     | 93.6     |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4170     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4180, episode_reward=124.00 +/- 18.13\n",
            "Episode length: 124.00 +/- 18.13\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 124      |\n",
            "|    mean_reward     | 124      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4180     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4190, episode_reward=108.60 +/- 40.84\n",
            "Episode length: 108.60 +/- 40.84\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 109      |\n",
            "|    mean_reward     | 109      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4190     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4200, episode_reward=120.40 +/- 46.48\n",
            "Episode length: 120.40 +/- 46.48\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 120      |\n",
            "|    mean_reward     | 120      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4200     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4210, episode_reward=187.80 +/- 153.91\n",
            "Episode length: 187.80 +/- 153.91\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 188      |\n",
            "|    mean_reward     | 188      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4210     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4220, episode_reward=123.40 +/- 63.49\n",
            "Episode length: 123.40 +/- 63.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 123      |\n",
            "|    mean_reward     | 123      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4220     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4230, episode_reward=128.60 +/- 62.62\n",
            "Episode length: 128.60 +/- 62.62\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 129      |\n",
            "|    mean_reward     | 129      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4230     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4240, episode_reward=170.60 +/- 165.18\n",
            "Episode length: 170.60 +/- 165.18\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 171      |\n",
            "|    mean_reward     | 171      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4240     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4250, episode_reward=151.60 +/- 54.26\n",
            "Episode length: 151.60 +/- 54.26\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 152      |\n",
            "|    mean_reward     | 152      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4250     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4260, episode_reward=100.60 +/- 33.92\n",
            "Episode length: 100.60 +/- 33.92\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 101      |\n",
            "|    mean_reward     | 101      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4260     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4270, episode_reward=163.40 +/- 169.07\n",
            "Episode length: 163.40 +/- 169.07\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 163      |\n",
            "|    mean_reward     | 163      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4270     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4280, episode_reward=95.60 +/- 13.17\n",
            "Episode length: 95.60 +/- 13.17\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 95.6     |\n",
            "|    mean_reward     | 95.6     |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4280     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4290, episode_reward=92.40 +/- 22.14\n",
            "Episode length: 92.40 +/- 22.14\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 92.4     |\n",
            "|    mean_reward     | 92.4     |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4290     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4300, episode_reward=141.00 +/- 39.84\n",
            "Episode length: 141.00 +/- 39.84\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 141      |\n",
            "|    mean_reward     | 141      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4300     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4310, episode_reward=118.00 +/- 68.77\n",
            "Episode length: 118.00 +/- 68.77\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 118      |\n",
            "|    mean_reward     | 118      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4310     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4320, episode_reward=99.80 +/- 20.43\n",
            "Episode length: 99.80 +/- 20.43\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 99.8     |\n",
            "|    mean_reward     | 99.8     |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4320     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4330, episode_reward=109.40 +/- 45.56\n",
            "Episode length: 109.40 +/- 45.56\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 109      |\n",
            "|    mean_reward     | 109      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4330     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4340, episode_reward=106.80 +/- 40.88\n",
            "Episode length: 106.80 +/- 40.88\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 107      |\n",
            "|    mean_reward     | 107      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4340     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4350, episode_reward=118.60 +/- 35.88\n",
            "Episode length: 118.60 +/- 35.88\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 119      |\n",
            "|    mean_reward     | 119      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4350     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4360, episode_reward=103.80 +/- 36.66\n",
            "Episode length: 103.80 +/- 36.66\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 104      |\n",
            "|    mean_reward     | 104      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4360     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4370, episode_reward=105.20 +/- 27.81\n",
            "Episode length: 105.20 +/- 27.81\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 105      |\n",
            "|    mean_reward     | 105      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4370     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4380, episode_reward=112.40 +/- 45.36\n",
            "Episode length: 112.40 +/- 45.36\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 112      |\n",
            "|    mean_reward     | 112      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4380     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4390, episode_reward=134.40 +/- 61.79\n",
            "Episode length: 134.40 +/- 61.79\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 134      |\n",
            "|    mean_reward     | 134      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4390     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4400, episode_reward=85.00 +/- 20.61\n",
            "Episode length: 85.00 +/- 20.61\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 85       |\n",
            "|    mean_reward     | 85       |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4400     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4410, episode_reward=103.40 +/- 6.86\n",
            "Episode length: 103.40 +/- 6.86\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 103      |\n",
            "|    mean_reward     | 103      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4410     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4420, episode_reward=107.20 +/- 30.96\n",
            "Episode length: 107.20 +/- 30.96\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 107      |\n",
            "|    mean_reward     | 107      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4420     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4430, episode_reward=92.20 +/- 18.37\n",
            "Episode length: 92.20 +/- 18.37\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 92.2     |\n",
            "|    mean_reward     | 92.2     |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4430     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4440, episode_reward=86.40 +/- 11.20\n",
            "Episode length: 86.40 +/- 11.20\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 86.4     |\n",
            "|    mean_reward     | 86.4     |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4440     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4450, episode_reward=186.60 +/- 157.34\n",
            "Episode length: 186.60 +/- 157.34\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 187      |\n",
            "|    mean_reward     | 187      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4450     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4460, episode_reward=115.20 +/- 51.49\n",
            "Episode length: 115.20 +/- 51.49\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 115      |\n",
            "|    mean_reward     | 115      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4460     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4470, episode_reward=84.80 +/- 14.47\n",
            "Episode length: 84.80 +/- 14.47\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 84.8     |\n",
            "|    mean_reward     | 84.8     |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4470     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4480, episode_reward=165.20 +/- 162.98\n",
            "Episode length: 165.20 +/- 162.98\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 165      |\n",
            "|    mean_reward     | 165      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4480     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4490, episode_reward=123.20 +/- 43.54\n",
            "Episode length: 123.20 +/- 43.54\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 123      |\n",
            "|    mean_reward     | 123      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4490     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4500, episode_reward=95.00 +/- 26.23\n",
            "Episode length: 95.00 +/- 26.23\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 95       |\n",
            "|    mean_reward     | 95       |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4500     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4510, episode_reward=109.00 +/- 23.77\n",
            "Episode length: 109.00 +/- 23.77\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 109      |\n",
            "|    mean_reward     | 109      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4510     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4520, episode_reward=118.00 +/- 17.60\n",
            "Episode length: 118.00 +/- 17.60\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 118      |\n",
            "|    mean_reward     | 118      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4520     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4530, episode_reward=109.40 +/- 42.51\n",
            "Episode length: 109.40 +/- 42.51\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 109      |\n",
            "|    mean_reward     | 109      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4530     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4540, episode_reward=94.00 +/- 22.46\n",
            "Episode length: 94.00 +/- 22.46\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 94       |\n",
            "|    mean_reward     | 94       |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4540     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4550, episode_reward=90.40 +/- 12.61\n",
            "Episode length: 90.40 +/- 12.61\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 90.4     |\n",
            "|    mean_reward     | 90.4     |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4550     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4560, episode_reward=219.60 +/- 151.82\n",
            "Episode length: 219.60 +/- 151.82\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 220      |\n",
            "|    mean_reward     | 220      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4560     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4570, episode_reward=99.40 +/- 24.93\n",
            "Episode length: 99.40 +/- 24.93\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 99.4     |\n",
            "|    mean_reward     | 99.4     |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4570     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4580, episode_reward=125.20 +/- 63.46\n",
            "Episode length: 125.20 +/- 63.46\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 125      |\n",
            "|    mean_reward     | 125      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4580     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4590, episode_reward=99.00 +/- 22.14\n",
            "Episode length: 99.00 +/- 22.14\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 99       |\n",
            "|    mean_reward     | 99       |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4590     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4600, episode_reward=89.60 +/- 23.01\n",
            "Episode length: 89.60 +/- 23.01\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 89.6     |\n",
            "|    mean_reward     | 89.6     |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4600     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4610, episode_reward=130.40 +/- 57.14\n",
            "Episode length: 130.40 +/- 57.14\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 130      |\n",
            "|    mean_reward     | 130      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4610     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4620, episode_reward=111.20 +/- 19.30\n",
            "Episode length: 111.20 +/- 19.30\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 111      |\n",
            "|    mean_reward     | 111      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4620     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4630, episode_reward=84.20 +/- 15.69\n",
            "Episode length: 84.20 +/- 15.69\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 84.2     |\n",
            "|    mean_reward     | 84.2     |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4630     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4640, episode_reward=113.00 +/- 34.33\n",
            "Episode length: 113.00 +/- 34.33\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 113      |\n",
            "|    mean_reward     | 113      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4640     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4650, episode_reward=98.40 +/- 9.02\n",
            "Episode length: 98.40 +/- 9.02\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 98.4     |\n",
            "|    mean_reward     | 98.4     |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4650     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4660, episode_reward=176.20 +/- 159.20\n",
            "Episode length: 176.20 +/- 159.20\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 176      |\n",
            "|    mean_reward     | 176      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4660     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4670, episode_reward=165.40 +/- 90.87\n",
            "Episode length: 165.40 +/- 90.87\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 165      |\n",
            "|    mean_reward     | 165      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4670     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4680, episode_reward=102.00 +/- 33.44\n",
            "Episode length: 102.00 +/- 33.44\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 102      |\n",
            "|    mean_reward     | 102      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4680     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4690, episode_reward=96.60 +/- 22.69\n",
            "Episode length: 96.60 +/- 22.69\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 96.6     |\n",
            "|    mean_reward     | 96.6     |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4690     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4700, episode_reward=95.80 +/- 21.19\n",
            "Episode length: 95.80 +/- 21.19\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 95.8     |\n",
            "|    mean_reward     | 95.8     |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4700     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4710, episode_reward=133.20 +/- 57.59\n",
            "Episode length: 133.20 +/- 57.59\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 133      |\n",
            "|    mean_reward     | 133      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4710     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4720, episode_reward=103.60 +/- 26.48\n",
            "Episode length: 103.60 +/- 26.48\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 104      |\n",
            "|    mean_reward     | 104      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4720     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4730, episode_reward=96.60 +/- 14.60\n",
            "Episode length: 96.60 +/- 14.60\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 96.6     |\n",
            "|    mean_reward     | 96.6     |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4730     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4740, episode_reward=109.00 +/- 25.23\n",
            "Episode length: 109.00 +/- 25.23\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 109      |\n",
            "|    mean_reward     | 109      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4740     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4750, episode_reward=132.60 +/- 59.16\n",
            "Episode length: 132.60 +/- 59.16\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 133      |\n",
            "|    mean_reward     | 133      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4750     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4760, episode_reward=133.40 +/- 57.01\n",
            "Episode length: 133.40 +/- 57.01\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 133      |\n",
            "|    mean_reward     | 133      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4760     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4770, episode_reward=142.00 +/- 50.71\n",
            "Episode length: 142.00 +/- 50.71\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 142      |\n",
            "|    mean_reward     | 142      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4770     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4780, episode_reward=119.80 +/- 50.13\n",
            "Episode length: 119.80 +/- 50.13\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 120      |\n",
            "|    mean_reward     | 120      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4780     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4790, episode_reward=89.40 +/- 17.58\n",
            "Episode length: 89.40 +/- 17.58\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 89.4     |\n",
            "|    mean_reward     | 89.4     |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4790     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4800, episode_reward=170.00 +/- 73.65\n",
            "Episode length: 170.00 +/- 73.65\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 170      |\n",
            "|    mean_reward     | 170      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4800     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4810, episode_reward=120.00 +/- 79.63\n",
            "Episode length: 120.00 +/- 79.63\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 120      |\n",
            "|    mean_reward     | 120      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4810     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4820, episode_reward=107.00 +/- 24.23\n",
            "Episode length: 107.00 +/- 24.23\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 107      |\n",
            "|    mean_reward     | 107      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4820     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4830, episode_reward=157.40 +/- 37.00\n",
            "Episode length: 157.40 +/- 37.00\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 157      |\n",
            "|    mean_reward     | 157      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4830     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4840, episode_reward=103.80 +/- 19.25\n",
            "Episode length: 103.80 +/- 19.25\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 104      |\n",
            "|    mean_reward     | 104      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4840     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4850, episode_reward=120.00 +/- 36.69\n",
            "Episode length: 120.00 +/- 36.69\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 120      |\n",
            "|    mean_reward     | 120      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4850     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4860, episode_reward=93.80 +/- 20.62\n",
            "Episode length: 93.80 +/- 20.62\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 93.8     |\n",
            "|    mean_reward     | 93.8     |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4860     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4870, episode_reward=172.60 +/- 91.44\n",
            "Episode length: 172.60 +/- 91.44\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 173      |\n",
            "|    mean_reward     | 173      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4870     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4880, episode_reward=98.20 +/- 31.50\n",
            "Episode length: 98.20 +/- 31.50\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 98.2     |\n",
            "|    mean_reward     | 98.2     |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4880     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4890, episode_reward=117.80 +/- 42.02\n",
            "Episode length: 117.80 +/- 42.02\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 118      |\n",
            "|    mean_reward     | 118      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4890     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4900, episode_reward=124.80 +/- 55.82\n",
            "Episode length: 124.80 +/- 55.82\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 125      |\n",
            "|    mean_reward     | 125      |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4900     |\n",
            "---------------------------------\n",
            "Eval num_timesteps=4910, episode_reward=89.60 +/- 17.78\n",
            "Episode length: 89.60 +/- 17.78\n",
            "---------------------------------\n",
            "| eval/              |          |\n",
            "|    mean_ep_length  | 89.6     |\n",
            "|    mean_reward     | 89.6     |\n",
            "| time/              |          |\n",
            "|    total_timesteps | 4910     |\n",
            "---------------------------------\n"
          ]
        },
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-8-3ea6369a5200>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mppo_expert\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mPPO\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'MlpPolicy'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0menv_id\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mverbose\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcreate_eval_env\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mppo_expert\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlearn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtotal_timesteps\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m3e4\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0meval_freq\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m10\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0mppo_expert\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msave\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"ppo_expert\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/stable_baselines3/ppo/ppo.py\u001b[0m in \u001b[0;36mlearn\u001b[0;34m(self, total_timesteps, callback, log_interval, eval_env, eval_freq, n_eval_episodes, tb_log_name, eval_log_path, reset_num_timesteps)\u001b[0m\n\u001b[1;32m    306\u001b[0m             \u001b[0mtb_log_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtb_log_name\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    307\u001b[0m             \u001b[0meval_log_path\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0meval_log_path\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 308\u001b[0;31m             \u001b[0mreset_num_timesteps\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mreset_num_timesteps\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    309\u001b[0m         )\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/stable_baselines3/common/on_policy_algorithm.py\u001b[0m in \u001b[0;36mlearn\u001b[0;34m(self, total_timesteps, callback, log_interval, eval_env, eval_freq, n_eval_episodes, tb_log_name, eval_log_path, reset_num_timesteps)\u001b[0m\n\u001b[1;32m    235\u001b[0m         \u001b[0;32mwhile\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnum_timesteps\u001b[0m \u001b[0;34m<\u001b[0m \u001b[0mtotal_timesteps\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    236\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 237\u001b[0;31m             \u001b[0mcontinue_training\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcollect_rollouts\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrollout_buffer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mn_rollout_steps\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mn_steps\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    238\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    239\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mcontinue_training\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/stable_baselines3/common/on_policy_algorithm.py\u001b[0m in \u001b[0;36mcollect_rollouts\u001b[0;34m(self, env, callback, rollout_buffer, n_rollout_steps)\u001b[0m\n\u001b[1;32m    182\u001b[0m             \u001b[0;31m# Give access to local variables\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    183\u001b[0m             \u001b[0mcallback\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupdate_locals\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlocals\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 184\u001b[0;31m             \u001b[0;32mif\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    185\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    186\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/stable_baselines3/common/callbacks.py\u001b[0m in \u001b[0;36mon_step\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     86\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnum_timesteps\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnum_timesteps\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     87\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 88\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_on_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     89\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     90\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mon_training_end\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/stable_baselines3/common/callbacks.py\u001b[0m in \u001b[0;36m_on_step\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    190\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mcallback\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcallbacks\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    191\u001b[0m             \u001b[0;31m# Return False (stop training) if at least one callback returns False\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 192\u001b[0;31m             \u001b[0mcontinue_training\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mon_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mcontinue_training\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    193\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mcontinue_training\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    194\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/stable_baselines3/common/callbacks.py\u001b[0m in \u001b[0;36mon_step\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     86\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnum_timesteps\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnum_timesteps\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     87\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 88\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_on_step\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     89\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     90\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mon_training_end\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/stable_baselines3/common/callbacks.py\u001b[0m in \u001b[0;36m_on_step\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    384\u001b[0m                 \u001b[0mreturn_episode_rewards\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    385\u001b[0m                 \u001b[0mwarn\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwarn\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 386\u001b[0;31m                 \u001b[0mcallback\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_log_success_callback\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    387\u001b[0m             )\n\u001b[1;32m    388\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/stable_baselines3/common/evaluation.py\u001b[0m in \u001b[0;36mevaluate_policy\u001b[0;34m(model, env, n_eval_episodes, deterministic, render, callback, reward_threshold, return_episode_rewards, warn)\u001b[0m\n\u001b[1;32m     83\u001b[0m     \u001b[0mstates\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     84\u001b[0m     \u001b[0;32mwhile\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mepisode_counts\u001b[0m \u001b[0;34m<\u001b[0m \u001b[0mepisode_count_targets\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0many\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 85\u001b[0;31m         \u001b[0mactions\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstates\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpredict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobservations\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstate\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mstates\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdeterministic\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdeterministic\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     86\u001b[0m         \u001b[0mobservations\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrewards\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdones\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minfos\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mactions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     87\u001b[0m         \u001b[0mcurrent_rewards\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0mrewards\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/stable_baselines3/common/base_class.py\u001b[0m in \u001b[0;36mpredict\u001b[0;34m(self, observation, state, mask, deterministic)\u001b[0m\n\u001b[1;32m    550\u001b[0m             \u001b[0;34m(\u001b[0m\u001b[0mused\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrecurrent\u001b[0m \u001b[0mpolicies\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    551\u001b[0m         \"\"\"\n\u001b[0;32m--> 552\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpolicy\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpredict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobservation\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstate\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmask\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdeterministic\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    553\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    554\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mset_random_seed\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mseed\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mint\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/stable_baselines3/common/policies.py\u001b[0m in \u001b[0;36mpredict\u001b[0;34m(self, observation, state, mask, deterministic)\u001b[0m\n\u001b[1;32m    334\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    335\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mth\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mno_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 336\u001b[0;31m             \u001b[0mactions\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_predict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobservation\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdeterministic\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdeterministic\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    337\u001b[0m         \u001b[0;31m# Convert to numpy\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    338\u001b[0m         \u001b[0mactions\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mactions\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcpu\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnumpy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/stable_baselines3/common/policies.py\u001b[0m in \u001b[0;36m_predict\u001b[0;34m(self, observation, deterministic)\u001b[0m\n\u001b[1;32m    626\u001b[0m         \u001b[0;34m:\u001b[0m\u001b[0;32mreturn\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mTaken\u001b[0m \u001b[0maction\u001b[0m \u001b[0maccording\u001b[0m \u001b[0mto\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mpolicy\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    627\u001b[0m         \"\"\"\n\u001b[0;32m--> 628\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_distribution\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobservation\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_actions\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdeterministic\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdeterministic\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    629\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    630\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mevaluate_actions\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mobs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mth\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTensor\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mactions\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mth\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mTuple\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mth\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTensor\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mth\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTensor\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mth\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTensor\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/stable_baselines3/common/policies.py\u001b[0m in \u001b[0;36mget_distribution\u001b[0;34m(self, obs)\u001b[0m\n\u001b[1;32m    655\u001b[0m         \u001b[0mfeatures\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mextract_features\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    656\u001b[0m         \u001b[0mlatent_pi\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmlp_extractor\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward_actor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfeatures\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 657\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_get_action_dist_from_latent\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlatent_pi\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    658\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    659\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mpredict_values\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mobs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mth\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mth\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTensor\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/stable_baselines3/common/policies.py\u001b[0m in \u001b[0;36m_get_action_dist_from_latent\u001b[0;34m(self, latent_pi)\u001b[0m\n\u001b[1;32m    606\u001b[0m         \u001b[0;32melif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maction_dist\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mCategoricalDistribution\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    607\u001b[0m             \u001b[0;31m# Here mean_actions are the logits before the softmax\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 608\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maction_dist\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mproba_distribution\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0maction_logits\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmean_actions\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    609\u001b[0m         \u001b[0;32melif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maction_dist\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mMultiCategoricalDistribution\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    610\u001b[0m             \u001b[0;31m# Here mean_actions are the flattened logits\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/stable_baselines3/common/distributions.py\u001b[0m in \u001b[0;36mproba_distribution\u001b[0;34m(self, action_logits)\u001b[0m\n\u001b[1;32m    272\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    273\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mproba_distribution\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maction_logits\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mth\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;34m\"CategoricalDistribution\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 274\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdistribution\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mCategorical\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlogits\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0maction_logits\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    275\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    276\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/distributions/categorical.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, probs, logits, validate_args)\u001b[0m\n\u001b[1;32m     58\u001b[0m                 \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"`logits` parameter must be at least one-dimensional.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     59\u001b[0m             \u001b[0;31m# Normalize\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 60\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlogits\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlogits\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mlogits\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlogsumexp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdim\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkeepdim\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     61\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_param\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprobs\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mprobs\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlogits\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     62\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_num_events\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_param\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hlyTfGAQ_Az1"
      },
      "source": [
        "check the performance of the trained agent"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-_rVEjC0_AQa",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "1873bf5d-c5f5-40b8-bf2b-59331a53fb73"
      },
      "source": [
        "mean_reward, std_reward = evaluate_policy(ppo_expert, env, n_eval_episodes=10)\n",
        "\n",
        "print(f\"Mean reward = {mean_reward} +/- {std_reward}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mean reward = 500.0 +/- 0.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S-Kv6v_V8aoJ"
      },
      "source": [
        "## Create Student\n",
        "\n",
        "We also create a student RL agent, which will later be trained with the expert dataset\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fLdLPUeC8aoL"
      },
      "source": [
        "a2c_student = A2C('MlpPolicy', env_id, verbose=1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sdW8_41-OcXn"
      },
      "source": [
        "# only valid for continuous actions\n",
        "# sac_student = SAC('MlpPolicy', env_id, verbose=1, policy_kwargs=dict(net_arch=[64, 64]))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m3GuNxcU8aoT"
      },
      "source": [
        "\n",
        "We now let our expert interact with the environment (except we already have expert data) and store resultant expert observations and actions to build an expert dataset.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "emodyZDW8aoU"
      },
      "source": [
        "num_interactions = int(4e4)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F3I_2s808aoZ"
      },
      "source": [
        "if isinstance(env.action_space, gym.spaces.Box):\n",
        "  expert_observations = np.empty((num_interactions,) + env.observation_space.shape)\n",
        "  expert_actions = np.empty((num_interactions,) + (env.action_space.shape[0],))\n",
        "\n",
        "else:\n",
        "  expert_observations = np.empty((num_interactions,) + env.observation_space.shape)\n",
        "  expert_actions = np.empty((num_interactions,) + env.action_space.shape)\n",
        "\n",
        "obs = env.reset()\n",
        "\n",
        "for i in tqdm(range(num_interactions)):\n",
        "    action, _ = ppo_expert.predict(obs, deterministic=True)\n",
        "    expert_observations[i] = obs\n",
        "    expert_actions[i] = action\n",
        "    obs, reward, done, info = env.step(action)\n",
        "    if done:\n",
        "        obs = env.reset()\n",
        "\n",
        "np.savez_compressed(\n",
        "    \"expert_data\",\n",
        "    expert_actions=expert_actions,\n",
        "    expert_observations=expert_observations,\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "toKEQE9i8aof"
      },
      "source": [
        "\n",
        "\n",
        "- To seamlessly use PyTorch in the training process, we subclass an `ExpertDataset` from PyTorch's base `Dataset`.\n",
        "- Note that we initialize the dataset with the previously generated expert observations and actions.\n",
        "- We further implement Python's `__getitem__` and `__len__` magic functions to allow PyTorch's dataset-handling to access arbitrary rows in the dataset and inform it about the length of the dataset.\n",
        "- For more information about PyTorch's datasets, you can read: https://pytorch.org/docs/stable/data.html.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qT72bR1i8aog"
      },
      "source": [
        "from torch.utils.data.dataset import Dataset, random_split"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xUetr5vl8aom"
      },
      "source": [
        "class ExpertDataSet(Dataset):\n",
        "    def __init__(self, expert_observations, expert_actions):\n",
        "        self.observations = expert_observations\n",
        "        self.actions = expert_actions\n",
        "        \n",
        "    def __getitem__(self, index):\n",
        "        return (self.observations[index], self.actions[index])\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.observations)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K9bNAhXp8aor"
      },
      "source": [
        "\n",
        "\n",
        "We now instantiate the `ExpertDataSet` and split it into training and test datasets.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lIdT-zMV8aot"
      },
      "source": [
        "expert_dataset = ExpertDataSet(expert_observations, expert_actions)\n",
        "\n",
        "train_size = int(0.8 * len(expert_dataset))\n",
        "\n",
        "test_size = len(expert_dataset) - train_size\n",
        "\n",
        "train_expert_dataset, test_expert_dataset = random_split(\n",
        "    expert_dataset, [train_size, test_size]\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_LgmtFFq8aox",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "outputId": "543280f3-84ff-440b-c2f1-a6dd3fe0f7e9"
      },
      "source": [
        "print(\"test_expert_dataset: \", len(test_expert_dataset))\n",
        "print(\"train_expert_dataset: \", len(train_expert_dataset))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "test_expert_dataset:  8000\n",
            "train_expert_dataset:  32000\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0v8PhG2r8ao4"
      },
      "source": [
        "\n",
        "\n",
        "NOTE: The supervised learning section of this code is adapted from: https://github.com/pytorch/examples/blob/master/mnist/main.py\n",
        "1. We extract the policy network of our RL student agent.\n",
        "2. We load the (labeled) expert dataset containing expert observations as inputs and expert actions as targets.\n",
        "3. We perform supervised learning, that is, we adjust the policy network's parameters such that given expert observations as inputs to the network, its outputs match the targets (expert actions).\n",
        "By training the policy network in this way the corresponding RL student agent is taught to behave like the expert agent that was used to created the expert dataset (Behavior Cloning).\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VwUhCTGU8ao5"
      },
      "source": [
        "def pretrain_agent(\n",
        "    student,\n",
        "    batch_size=64,\n",
        "    epochs=1000,\n",
        "    scheduler_gamma=0.7,\n",
        "    learning_rate=1.0,\n",
        "    log_interval=100,\n",
        "    no_cuda=True,\n",
        "    seed=1,\n",
        "    test_batch_size=64,\n",
        "):\n",
        "    use_cuda = not no_cuda and th.cuda.is_available()\n",
        "    th.manual_seed(seed)\n",
        "    device = th.device(\"cuda\" if use_cuda else \"cpu\")\n",
        "    kwargs = {\"num_workers\": 1, \"pin_memory\": True} if use_cuda else {}\n",
        "\n",
        "    if isinstance(env.action_space, gym.spaces.Box):\n",
        "      criterion = nn.MSELoss()\n",
        "    else:\n",
        "      criterion = nn.CrossEntropyLoss()\n",
        "\n",
        "    # Extract initial policy\n",
        "    model = student.policy.to(device)\n",
        "\n",
        "    def train(model, device, train_loader, optimizer):\n",
        "        model.train()\n",
        "\n",
        "        for batch_idx, (data, target) in enumerate(train_loader):\n",
        "            data, target = data.to(device), target.to(device)\n",
        "            optimizer.zero_grad()\n",
        "\n",
        "            if isinstance(env.action_space, gym.spaces.Box):\n",
        "              # A2C/PPO policy outputs actions, values, log_prob\n",
        "              # SAC/TD3 policy outputs actions only\n",
        "              if isinstance(student, (A2C, PPO)):\n",
        "                action, _, _ = model(data)\n",
        "              else:\n",
        "                # SAC/TD3:\n",
        "                action = model(data)\n",
        "              action_prediction = action.double()\n",
        "            else:\n",
        "              # Retrieve the logits for A2C/PPO when using discrete actions\n",
        "              latent_pi, _, _ = model._get_latent(data)\n",
        "              logits = model.action_net(latent_pi)\n",
        "              action_prediction = logits\n",
        "              target = target.long()\n",
        "\n",
        "            loss = criterion(action_prediction, target)\n",
        "            loss.backward()\n",
        "            optimizer.step()\n",
        "            if batch_idx % log_interval == 0:\n",
        "                print(\n",
        "                    \"Train Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}\".format(\n",
        "                        epoch,\n",
        "                        batch_idx * len(data),\n",
        "                        len(train_loader.dataset),\n",
        "                        100.0 * batch_idx / len(train_loader),\n",
        "                        loss.item(),\n",
        "                    )\n",
        "                )\n",
        "    def test(model, device, test_loader):\n",
        "        model.eval()\n",
        "        test_loss = 0\n",
        "        with th.no_grad():\n",
        "            for data, target in test_loader:\n",
        "                data, target = data.to(device), target.to(device)\n",
        "\n",
        "                if isinstance(env.action_space, gym.spaces.Box):\n",
        "                  # A2C/PPO policy outputs actions, values, log_prob\n",
        "                  # SAC/TD3 policy outputs actions only\n",
        "                  if isinstance(student, (A2C, PPO)):\n",
        "                    action, _, _ = model(data)\n",
        "                  else:\n",
        "                    # SAC/TD3:\n",
        "                    action = model(data)\n",
        "                  action_prediction = action.double()\n",
        "                else:\n",
        "                  # Retrieve the logits for A2C/PPO when using discrete actions\n",
        "                  latent_pi, _, _ = model._get_latent(data)\n",
        "                  logits = model.action_net(latent_pi)\n",
        "                  action_prediction = logits\n",
        "                  target = target.long()\n",
        "\n",
        "                test_loss = criterion(action_prediction, target)\n",
        "        test_loss /= len(test_loader.dataset)\n",
        "        print(f\"Test set: Average loss: {test_loss:.4f}\")\n",
        "\n",
        "    # Here, we use PyTorch `DataLoader` to our load previously created `ExpertDataset` for training\n",
        "    # and testing\n",
        "    train_loader = th.utils.data.DataLoader(\n",
        "        dataset=train_expert_dataset, batch_size=batch_size, shuffle=True, **kwargs\n",
        "    )\n",
        "    test_loader = th.utils.data.DataLoader(\n",
        "        dataset=test_expert_dataset, batch_size=test_batch_size, shuffle=True, **kwargs,\n",
        "    )\n",
        "\n",
        "    # Define an Optimizer and a learning rate schedule.\n",
        "    optimizer = optim.Adadelta(model.parameters(), lr=learning_rate)\n",
        "    scheduler = StepLR(optimizer, step_size=1, gamma=scheduler_gamma)\n",
        "\n",
        "    # Now we are finally ready to train the policy model.\n",
        "    for epoch in range(1, epochs + 1):\n",
        "        train(model, device, train_loader, optimizer)\n",
        "        test(model, device, test_loader)\n",
        "        scheduler.step()\n",
        "\n",
        "    # Implant the trained policy network back into the RL student agent\n",
        "    a2c_student.policy = model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nkEP6i0hEu_R"
      },
      "source": [
        "Evaluate the agent before pretraining, it should be random"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_7kvYIneEui8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "fb450cd6-ec13-4cd6-bb14-abc12c42c775"
      },
      "source": [
        "mean_reward, std_reward = evaluate_policy(a2c_student, env, n_eval_episodes=10)\n",
        "\n",
        "print(f\"Mean reward = {mean_reward} +/- {std_reward}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mean reward = 87.7 +/- 42.99546487712396\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SgduZAbF8ao9"
      },
      "source": [
        "\n",
        "\n",
        "Having defined the training procedure we can now run the pretraining!\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eI1EFFnW8ao-"
      },
      "source": [
        "pretrain_agent(\n",
        "    a2c_student,\n",
        "    epochs=3,\n",
        "    scheduler_gamma=0.7,\n",
        "    learning_rate=1.0,\n",
        "    log_interval=100,\n",
        "    no_cuda=True,\n",
        "    seed=1,\n",
        "    batch_size=64,\n",
        "    test_batch_size=1000,\n",
        ")\n",
        "a2c_student.save(\"a2c_student\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RK3Q5Jm58apE"
      },
      "source": [
        "\n",
        "\n",
        "Finally, let us test how well our RL agent student learned to mimic the behavior of the expert\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GKZ8O--m8apF",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "398abb60-b1ac-43b1-cb40-f9045b8b2339"
      },
      "source": [
        "mean_reward, std_reward = evaluate_policy(a2c_student, env, n_eval_episodes=10)\n",
        "\n",
        "print(f\"Mean reward = {mean_reward} +/- {std_reward}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mean reward = 500.0 +/- 0.0\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}