The_Picnic_Vision_Submission.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "The_Picnic_Vision_Submission.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "metadata": {
        "id": "4-Zcu-kDZS9j",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Picnic Hackathon 🥇🏆💯\n",
        " **When great customer support meets data**\n",
        "\n",
        "\n",
        "## Why the challenge?\n",
        "One of our core beliefs is to offer our customers the best support possible, by allowing them, for example, to send in pictures of defect products they wish to be reimbursed for. But processing these pictures is very time-consuming as it is all done manually.\n",
        "\n",
        "## What is the challenge?\n",
        "The challenge we propose is the following: As a first step in helping customer support, come up with a way of labeling every picture that comes in according to the product that is in the picture. To keep with the Picnic spirit, we encourage to be as innovative and creative with your solutions as possible.\n"
      ]
    },
    {
      "metadata": {
        "id": "r9UfbZyghP68",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## This NoteBook 📓📒\n",
        "In this notebook we will show you how to reproduce the result I got in my **Final Submission**.\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "Tm-OZXKJiBIc",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Requirments ✅\n",
        "\n",
        "- Python 3\n",
        "- Fastai library\n",
        "- Pandas & Numpy\n",
        "\n",
        "I have used Colab Enviroment during this hackathon, because it offer free GPU's, and also for the great compatibility with Google Drive, which was the holder of the Images for training and Testing, and for the Final Model Weights.\n",
        "\n",
        "The Final Model File can be access via this link, make sure you change the path in the section of **Loading The Solution Model** and change it to what you have, I'm using Colab so I will access it directly after getting reference to my drive.\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "qXgWqei1ZcL0",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Setting & Imports\n",
        "Here we load all the library required."
      ]
    },
    {
      "metadata": {
        "id": "2yBuA-ZaZd8r",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Data Science Things\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "\n",
        "# fast.ai Library\n",
        "import fastai\n",
        "from fastai.vision import *\n",
        "from fastai.vision.models import *\n",
        "import torch\n",
        "\n",
        "# Images & Paths\n",
        "from PIL import ImageFile\n",
        "from pathlib import Path\n",
        "import glob\n",
        "\n",
        "#other\n",
        "from google.colab import drive\n",
        "from datetime import date"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Nr99TyCjZrIL",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# setting random seed\n",
        "np.random.seed(42)\n",
        "# make sure to change this to what you have, this path will be used for loading test images.\n",
        "path_to_folder = 'gdrive/My Drive/Dataset/The Picnic Hackathon 2019/'\n",
        "ImageFile.LOAD_TRUNCATED_IMAGES = True"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "9xyoqVv3ZyXj",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# getting reference to our model file from drive.\n",
        "drive.mount('/content/gdrive')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "XDxa4uPYbFih",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Loading The Solution Model\n",
        "In this section, We will load the trained model file, which got about 0.86 as F1-Score, make sur you can reference the model file, otherwise contact me. I have tried a lot of pre-trained model, the one which got me to the last submission I've sent was DenseNet161, here is the [Officiel Repository](https://github.com/liuzhuang13/DenseNet) and It's [Paper](https://arxiv.org/pdf/1608.06993v3.pdf) "
      ]
    },
    {
      "metadata": {
        "id": "q06QNQrUbaaS",
        "colab_type": "code",
        "outputId": "2083735c-bcf7-4cb6-ab9a-eb55eb454d16",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "# sometimes this cell cause error due Google Drive OSError, just re-run it, it happen only once.\n",
        "# change this to what it convient for you. Where the model file is ?\n",
        "path_to_model_file = 'gdrive/My Drive/'\n",
        "# change this if you have renamed the file. What is the name of the file ?\n",
        "file_name = 'densenet161_final_model.pkl'\n",
        "model = load_learner(path = path_to_model_file, file = file_name)\n",
        "print('Done')"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Done\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "TTiGyA6nblgI",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Predicting the Test Set."
      ]
    },
    {
      "metadata": {
        "id": "C4uChmuAblFQ",
        "colab_type": "code",
        "outputId": "0cd3bd75-15e4-4a76-b3f3-3c4a18adf497",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "# first we get reference to all the fiels in the test set.\n",
        "files = glob.glob(path_to_folder + 'test/*')\n",
        "total = len(files)\n",
        "print('Found {} images'.format(total))"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Found 820 images\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "IVrqy6Z5beJ1",
        "colab_type": "code",
        "outputId": "a6bff654-51d6-4442-8e9b-8bba59cf0252",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "# Lopping over all the file, load -> predict -> and Store the results.\n",
        "# final array to hold the results.\n",
        "results = []\n",
        "# variable to track the progress.\n",
        "i = 1\n",
        "\n",
        "for file in files:  \n",
        "    print(\"\\rImage #{} of {} , Total Progress {}% .\".format(i, total, int((i/total)*100)), end=\"\")\n",
        "    sys.stdout.flush()\n",
        "    i+=1\n",
        "    # open the image\n",
        "    img = open_image(Path(file)).apply_tfms(None, size = 224)\n",
        "    # predict\n",
        "    predicted_class, idx, out = model.predict(img)\n",
        "    # getting file name.\n",
        "    filename = os.path.basename(file)\n",
        "    results.append([filename, str(predicted_class)])"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Image #820 of 820 , Total Progress 100% ."
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "KgJYBboUeAHK",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Constructing The Submission file.\n",
        "headers = ['file', 'label']\n",
        "submission = pd.DataFrame(results, columns=headers)\n",
        "submission = submission.sort_values(['file'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Lm5cdXrYk23_",
        "colab_type": "code",
        "outputId": "19dee253-aa59-45c7-d403-4baa056340fc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "cell_type": "code",
      "source": [
        "# Make sure the right appearance\n",
        "submission.head()"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>file</th>\n",
              "      <th>label</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>503</th>\n",
              "      <td>7263.jpeg</td>\n",
              "      <td>Bell peppers, zucchinis &amp; eggplants</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>520</th>\n",
              "      <td>7264.jpeg</td>\n",
              "      <td>Eggs</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>571</th>\n",
              "      <td>7265.jpeg</td>\n",
              "      <td>Broccoli, cauliflowers, carrots &amp; radish</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>332</th>\n",
              "      <td>7266.png</td>\n",
              "      <td>Lunch &amp; Deli Meats</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>513</th>\n",
              "      <td>7267.jpeg</td>\n",
              "      <td>Potatoes</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "          file                                     label\n",
              "503  7263.jpeg       Bell peppers, zucchinis & eggplants\n",
              "520  7264.jpeg                                      Eggs\n",
              "571  7265.jpeg  Broccoli, cauliflowers, carrots & radish\n",
              "332   7266.png                        Lunch & Deli Meats\n",
              "513  7267.jpeg                                  Potatoes"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "metadata": {
        "id": "hN1ciyGIeHFV",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# saving the file into the desired format.\n",
        "today = date.today()\n",
        "name_file = today.strftime(\"%d-%m-%y\") + '_1.tsv'\n",
        "submission.to_csv(name_file, sep = '\\t', index = False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "6R6dzSL9oDU9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Conclusion 😀🎉🙋‍♂️\n",
        "I want to thanks the Picnic Company for this opprtunity to tackle real world problem, sharing their problem with the community of Devpost Hackers and Letting us Expirements lot fo things on their dataset."
      ]
    }
  ]
}