RL_Test1.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "Q-Learning with FrozenLake-v1 ⛄\n"
      ],
      "metadata": {
        "id": "1Vxpq654EWin"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Step 0: Setup a Virtual Display 💻\n",
        "During the notebook, we'll need to generate a replay video. To do so, with colab, we need to have a virtual screen to be able to render the environment (and thus record the frames).\n",
        "\n",
        "Hence the following cell will install virtual screen libraries and create and run a virtual screen 🖥"
      ],
      "metadata": {
        "id": "MYXITT6OEhkQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%capture\n",
        "!pip install pyglet==1.5.1 \n",
        "!apt install python-opengl\n",
        "!apt install ffmpeg\n",
        "!apt install xvfb\n",
        "!pip3 install pyvirtualdisplay\n",
        "\n",
        "# Virtual display\n",
        "from pyvirtualdisplay import Display\n",
        "\n",
        "virtual_display = Display(visible=0, size=(1400, 900))\n",
        "virtual_display.start()"
      ],
      "metadata": {
        "id": "Ess_eYMQEfKU"
      },
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Step 1: Install dependencies 🔽\n",
        "The first step is to install the dependencies, we’ll install multiple ones:\n",
        "\n",
        "gym: Contains the FrozenLake-v1 ⛄ and Taxi-v3 🚕 environments.\n",
        "\n",
        "pygame: Used for the FrozenLake-v1 and Taxi-v3 UI.\n",
        "\n",
        "numPy: Used for handling our Q-table."
      ],
      "metadata": {
        "id": "n5TKHhqpIKha"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%capture\n",
        "!pip install gym==0.24 # We install the newest gym version for the Taxi-v3 \"rgb_array version\"\n",
        "!pip install pygame\n",
        "!pip install numpy\n",
        "\n",
        "!pip install huggingface_hub\n",
        "!pip install pickle5\n",
        "!pip install pyyaml==6.0 # avoid key error metadata\n",
        "!pip install imageio imageio_ffmpeg"
      ],
      "metadata": {
        "id": "C14zNqJdIMV_"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Step 2: Import the packages 📦\n",
        "In addition to the installed libraries, we also use:\n",
        "\n",
        "random: To generate random numbers (that will be useful for Epsilon-Greedy Policy).\n",
        "\n",
        "imageio: To generate a replay video"
      ],
      "metadata": {
        "id": "Jfh1zdZMJVii"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import gym\n",
        "import random\n",
        "import imageio\n",
        "import os\n",
        "\n",
        "import pickle5 as pickle"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "POf2WUXaJYOD",
        "outputId": "102f2cd5-933f-4a7a-def6-f666d3d45d85"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Warning: Gym version v0.24.0 has a number of critical issues with `gym.make` such that the `reset` and `step` functions are called before returning the environment. It is recommend to downgrading to v0.23.1 or upgrading to v0.25.1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Part 1: Frozen Lake ⛄"
      ],
      "metadata": {
        "id": "Q3g_iaj0Lwhi"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We're going to train our Q-Learning agent to navigate from the starting state (S) to the goal state (G) by walking only on frozen tiles (F) and avoid holes (H).\n",
        "\n",
        "We can have two sizes of environment:\n",
        "\n",
        "map_name=\"4x4\": a 4x4 grid version\n",
        "\n",
        "map_name=\"8x8\": a 8x8 grid version\n",
        "\n",
        "The environment has two modes:\n",
        "\n",
        "is_slippery=False: The agent always move in the intended direction due to the non-slippery nature of the frozen lake.\n",
        "\n",
        "is_slippery=True: The agent may not always move in the intended direction due to the slippery nature of the frozen lake (stochastic)."
      ],
      "metadata": {
        "id": "2ZswVr12L5Xy"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "For now let's keep it simple with the 4x4 map and non-slippery"
      ],
      "metadata": {
        "id": "34389LjlM50q"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create the FrozenLake-v1 environment using 4x4 map and non-slippery version\n",
        "env = gym.make(\"FrozenLake-v1\",map_name=\"4x4\",is_slippery=False)"
      ],
      "metadata": {
        "id": "DndsA3P-L-G7"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's see what the Environment looks like:"
      ],
      "metadata": {
        "id": "bbXz0EVQNNHc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# We create our environment with gym.make(\"<name_of_the_environment>\")\n",
        "env.reset()\n",
        "print(\"_____OBSERVATION SPACE_____ \\n\")\n",
        "print(\"Observation Space\", env.observation_space)\n",
        "print(\"Sample observation\", env.observation_space.sample()) # Get a random observation"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "f_tjoWxdNN6a",
        "outputId": "8663f6cd-8d64-4a76-8c02-649ce649a921"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "_____OBSERVATION SPACE_____ \n",
            "\n",
            "Observation Space Discrete(16)\n",
            "Sample observation 14\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"\\n _____ACTION SPACE_____ \\n\")\n",
        "print(\"Action Space Shape\", env.action_space.n)\n",
        "print(\"Action Space Sample\", env.action_space.sample()) # Take a random action"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wnAFErW4Odtp",
        "outputId": "b4c18c9c-58ab-4aef-bd39-3535f066faf7"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            " _____ACTION SPACE_____ \n",
            "\n",
            "Action Space Shape 4\n",
            "Action Space Sample 2\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The action space (the set of possible actions the agent can take) is discrete with 4 actions available 🎮:\n",
        "\n",
        "0: GO LEFT\n",
        "\n",
        "1: GO DOWN\n",
        "\n",
        "2: GO RIGHT\n",
        "\n",
        "3: GO UP\n",
        "\n",
        "Reward function 💰:\n",
        "\n",
        "Reach goal: +1\n",
        "\n",
        "Reach hole: 0\n",
        "\n",
        "Reach frozen: 0"
      ],
      "metadata": {
        "id": "Lt9h1ciKiQ-W"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Step 2: Create and Initialize the Q-table 🗄️"
      ],
      "metadata": {
        "id": "FraD_Cg3iWTx"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "It's time to initialize our Q-table! To know how many rows (states) and columns (actions) to use, we need to know the action and observation space. OpenAI Gym provides us a way to do that: env.action_space.n and env.observation_space.n"
      ],
      "metadata": {
        "id": "6ay4VE1riyU0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "state_space = env.observation_space.n\n",
        "print(\"There are \", state_space, \" possible states\")\n",
        "\n",
        "action_space = env.action_space.n\n",
        "print(\"There are \", action_space, \" possible actions\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Kw4ODbLhil4y",
        "outputId": "41c14efa-5bd6-470e-8cd6-b486b1431cc9"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "There are  16  possible states\n",
            "There are  4  possible actions\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Let's create our Qtable of size (state_space, action_space) and initialized each values at 0 using np.zeros\n",
        "def initialize_q_table(state_space, action_space):\n",
        "  Qtable = np.zeros((state_space,action_space))\n",
        "  return Qtable"
      ],
      "metadata": {
        "id": "qR8J2D03i8lB"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "Qtable_frozenlake = initialize_q_table(state_space, action_space)"
      ],
      "metadata": {
        "id": "oNQZzNsMhLM6"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Step 3: Define the epsilon-greedy policy 🤖\n",
        "Epsilon-Greedy is the training policy that handles the exploration/exploitation trade-off.\n",
        "\n",
        "The idea with Epsilon Greedy:\n",
        "\n",
        "With probability 1 - ɛ : we do exploitation (aka our agent selects the action with the highest state-action pair value).\n",
        "\n",
        "With probability ɛ: we do exploration (trying random action).\n",
        "\n",
        "And as the training goes, we progressively reduce the epsilon value since we will need less and less exploration and more exploitation."
      ],
      "metadata": {
        "id": "KizCpAyxjO5E"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def epsilon_greedy_policy(Qtable, state, epsilon):\n",
        "  # Randomly generate a number between 0 and 1\n",
        "  random_num = random.uniform(0,1)\n",
        "  # if random_num > greater than epsilon --> exploitation\n",
        "  if random_num > epsilon:\n",
        "    # Take the action with the highest value given a state\n",
        "    # np.argmax can be useful here\n",
        "    action = np.argmax(Qtable[state])\n",
        "  # else --> exploration\n",
        "  else:\n",
        "    action = env.action_space.sample() # Take a random action\n",
        "  \n",
        "  return action"
      ],
      "metadata": {
        "id": "xA3mY2T4jQtf"
      },
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Step 4: Define the greedy policy 🤖\n",
        "\n",
        "Remember we have two policies since Q-Learning is an off-policy algorithm. This means we're using a different policy for acting and updating the value function.\n",
        "\n",
        "\n",
        "Epsilon greedy policy (acting policy)\n",
        "\n",
        "Greedy policy (updating policy)\n",
        "\n",
        "Greedy policy will also be the final policy we'll have when the Q-learning agent will be trained. The greedy policy is used to select an action from the Q-table."
      ],
      "metadata": {
        "id": "aqb8wdV7nQrM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def greedy_policy(Qtable, state):\n",
        "  # Exploitation: take the action with the highest state, action value\n",
        "  action = np.argmax(Qtable[state])\n",
        "  \n",
        "  return action"
      ],
      "metadata": {
        "id": "m8fZf_iLnWhF"
      },
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Step 5: Define the hyperparameters ⚙️\n",
        "\n",
        "The exploration related hyperparamters are some of the most important ones.\n",
        "\n",
        "We need to make sure that our agent explores enough the state space in order to learn a good value approximation, in order to do that we need to have progressive decay of the epsilon.\n",
        "\n",
        "If you decrease too fast epsilon (too high decay_rate), you take the risk that your agent is stuck, since your agent didn't explore enough the state space and hence can't solve the problem."
      ],
      "metadata": {
        "id": "tY4VMFYAoVic"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Training parameters\n",
        "n_training_episodes = 10000  # Total training episodes\n",
        "learning_rate = 0.7          # Learning rate\n",
        "\n",
        "# Evaluation parameters\n",
        "n_eval_episodes = 100        # Total number of test episodes\n",
        "\n",
        "# Environment parameters\n",
        "env_id = \"FrozenLake-v1\"     # Name of the environment\n",
        "max_steps = 99               # Max steps per episode\n",
        "gamma = 0.95                 # Discounting rate\n",
        "eval_seed = []               # The evaluation seed of the environment\n",
        "\n",
        "# Exploration parameters\n",
        "epsilon = 1.0                 # Exploration rate\n",
        "max_epsilon = 1.0             # Exploration probability at start\n",
        "min_epsilon = 0.05            # Minimum exploration probability \n",
        "decay_rate = 0.0005            # Exponential decay rate for exploration prob"
      ],
      "metadata": {
        "id": "bDxROIRWoY4y"
      },
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Step 6: Create the training loop method"
      ],
      "metadata": {
        "id": "C5qfJUpsowun"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def train(n_training_episodes, min_epsilon, max_epsilon, decay_rate, env, max_steps, Qtable):\n",
        "  for episode in range(n_training_episodes):\n",
        "    # Reduce epsilon (because we need less and less exploration)\n",
        "    epsilon = min_epsilon + (max_epsilon - min_epsilon)*np.exp(-decay_rate*episode)\n",
        "    # Reset the environment\n",
        "    state = env.reset()\n",
        "    step = 0\n",
        "    done = False\n",
        "\n",
        "    # repeat\n",
        "    for step in range(max_steps):\n",
        "      # Choose the action At using epsilon greedy policy\n",
        "      action = epsilon_greedy_policy(Qtable, state, epsilon)\n",
        "\n",
        "      # Take action At and observe Rt+1 and St+1\n",
        "      # Take the action (a) and observe the outcome state(s') and reward (r)\n",
        "      new_state, reward, done, info = env.step(action)\n",
        "\n",
        "      # Update Q(s,a):= Q(s,a) + lr [R(s,a) + gamma * max Q(s',a') - Q(s,a)]\n",
        "      Qtable[state][action] = Qtable[state][action] + learning_rate * (reward + gamma * np.max(Qtable[new_state]) - Qtable[state][action]) \n",
        "\n",
        "      # If done, finish the episode\n",
        "      if done:\n",
        "        break\n",
        "      \n",
        "      # Our state is the new state\n",
        "      state = new_state\n",
        "  return Qtable"
      ],
      "metadata": {
        "id": "hzdNo7NEoxp3"
      },
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Step 7: Train the Q-Learning agent 🏃"
      ],
      "metadata": {
        "id": "9hB9QsYLqxYI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "Qtable_frozenlake = train(n_training_episodes, min_epsilon, max_epsilon, decay_rate, env, max_steps, Qtable_frozenlake)"
      ],
      "metadata": {
        "id": "zQ985kfSnR0H"
      },
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Step 8: Let's see what our Q-Learning table looks like now 👀"
      ],
      "metadata": {
        "id": "wnPInrmBrDUF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "Qtable_frozenlake"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HiIJBcoGrEVU",
        "outputId": "82ebaf1c-6b4f-480a-fdb6-b3b34e44b9b9"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.73509189, 0.77378094, 0.77378094, 0.73509189],\n",
              "       [0.73509189, 0.        , 0.81450625, 0.77378094],\n",
              "       [0.77378094, 0.857375  , 0.77378094, 0.81450625],\n",
              "       [0.81450625, 0.        , 0.77378094, 0.77378094],\n",
              "       [0.77378094, 0.81450625, 0.        , 0.73509189],\n",
              "       [0.        , 0.        , 0.        , 0.        ],\n",
              "       [0.        , 0.9025    , 0.        , 0.81450625],\n",
              "       [0.        , 0.        , 0.        , 0.        ],\n",
              "       [0.81450625, 0.        , 0.857375  , 0.77378094],\n",
              "       [0.81450625, 0.9025    , 0.9025    , 0.        ],\n",
              "       [0.857375  , 0.95      , 0.        , 0.857375  ],\n",
              "       [0.        , 0.        , 0.        , 0.        ],\n",
              "       [0.        , 0.        , 0.        , 0.        ],\n",
              "       [0.        , 0.9025    , 0.95      , 0.857375  ],\n",
              "       [0.9025    , 0.95      , 1.        , 0.9025    ],\n",
              "       [0.        , 0.        , 0.        , 0.        ]])"
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Step 9: Define the evaluation method 📝"
      ],
      "metadata": {
        "id": "m1yoM87srOJL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def evaluate_agent(env, max_steps, n_eval_episodes, Q, seed):\n",
        "  \"\"\"\n",
        "  Evaluate the agent for ``n_eval_episodes`` episodes and returns average reward and std of reward.\n",
        "  :param env: The evaluation environment\n",
        "  :param n_eval_episodes: Number of episode to evaluate the agent\n",
        "  :param Q: The Q-table\n",
        "  :param seed: The evaluation seed array (for taxi-v3)\n",
        "  \"\"\"\n",
        "  episode_rewards = []\n",
        "  for episode in range(n_eval_episodes):\n",
        "    if seed:\n",
        "      state = env.reset(seed=seed[episode])\n",
        "    else:\n",
        "      state = env.reset()\n",
        "    step = 0\n",
        "    done = False\n",
        "    total_rewards_ep = 0\n",
        "    \n",
        "    for step in range(max_steps):\n",
        "      # Take the action (index) that have the maximum expected future reward given that state\n",
        "      action = np.argmax(Q[state][:])\n",
        "      new_state, reward, done, info = env.step(action)\n",
        "      total_rewards_ep += reward\n",
        "        \n",
        "      if done:\n",
        "        break\n",
        "      state = new_state\n",
        "    episode_rewards.append(total_rewards_ep)\n",
        "  mean_reward = np.mean(episode_rewards)\n",
        "  std_reward = np.std(episode_rewards)\n",
        "\n",
        "  return mean_reward, std_reward"
      ],
      "metadata": {
        "id": "JraCJ9ANrPF0"
      },
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Step 10: Evaluate our Q-Learning agent 📈\n",
        "\n",
        "Normally you should have mean reward of 1.0\n",
        "\n",
        "It's relatively easy since the state space is really small (16). What you can try to do is to replace with the slippery version.\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "5zw0vkLe4J36"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Evaluate our Agent\n",
        "mean_reward, std_reward = evaluate_agent(env, max_steps, n_eval_episodes, Qtable_frozenlake, eval_seed)\n",
        "print(f\"Mean_reward={mean_reward:.2f} +/- {std_reward:.2f}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PTzMMBfO4Ks3",
        "outputId": "1187a6a8-f66d-4781-b341-fba908fe7d1c"
      },
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mean_reward=1.00 +/- 0.00\n"
          ]
        }
      ]
    }
  ]
}