book/NLP_chapter1.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyPLr1UqN1+B6uC8bOPhcg7t",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/giorgiosld/Natural-Language-Processing/blob/main/book_exercises/NLP_exercise.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Computing with Language: Texts and Words\n"
      ],
      "metadata": {
        "id": "uW401WJzbiX0"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Getting Started with NLTK\n",
        "Import NLTK and download NLTK Book Collection."
      ],
      "metadata": {
        "id": "N8G2YXE6_p-7"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ldB-JfSWLZGs"
      },
      "outputs": [],
      "source": [
        "import nltk\n",
        "nltk.download()"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "See insight data."
      ],
      "metadata": {
        "id": "EBafLa3b_oA8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from nltk.book import *"
      ],
      "metadata": {
        "collapsed": true,
        "id": "3gBwne4sAEDU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Search for concordance words inside the data."
      ],
      "metadata": {
        "id": "G4tQ0A2mCvKW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# print(text1)\n",
        "print(text1.concordance(\"monstrous\"))\n",
        "# print(text2.concordance(\"affection\"))\n",
        "# print(text2.concordance(\"lived\"))"
      ],
      "metadata": {
        "collapsed": true,
        "id": "50Fxtv7bAfxV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Since the concordance permits to see words in context is possible find out words appear in a similar rangd of contexts."
      ],
      "metadata": {
        "id": "MX94uvv2DgG6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "text1.similar(\"monstrous\")\n",
        "text2.similar(\"monstrous\")"
      ],
      "metadata": {
        "id": "ism9lEMrDwTn"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "It was seen that we get different results for different texts because it dependens on connotation. For this example *monstrous* has a positive connotation similar to the intensifier word *very*.\n",
        "But we can do more examining just the contexts shared by two or more words."
      ],
      "metadata": {
        "id": "aYTC9_hIEDrm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "text2.common_contexts([\"monstrous\", \"very\"])"
      ],
      "metadata": {
        "id": "nrvzC9YAE5hH"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can also determine the *location* of a word in the text. Using a particular plot named **disperion plot**."
      ],
      "metadata": {
        "id": "7kPDRlK_Fyzm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "text4.dispersion_plot([\"citizens\", \"democracy\", \"freedom\", \"duties\", \"America\"])"
      ],
      "metadata": {
        "id": "43f28HFwF8xS"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Is also possible generate some random text in the various styles."
      ],
      "metadata": {
        "id": "fKoYNd0AGjv1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "text3.generate()"
      ],
      "metadata": {
        "id": "klfLFq9MGqXd"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The concept of **token** is way to represent a sequence of charcters treated as a group. It is possible to see how many tokens are in a specific document."
      ],
      "metadata": {
        "id": "v-Us5DZvIqPd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(len(text3))\n",
        "print(sorted(set(text3)))\n",
        "print(len(set(text3)))"
      ],
      "metadata": {
        "id": "z8c-FxszJEqy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Through the next commands it's possible to have an idea of the richness of text."
      ],
      "metadata": {
        "id": "RJFtxahhJXyJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "len(set(text3)) / len(text3)"
      ],
      "metadata": {
        "id": "WqbceVU0Jo9x"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Is also possible to see the number of occurencies and the percentage of the text is taken up by a specific word."
      ],
      "metadata": {
        "id": "K2tCc9CTJxPp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "text3.count(\"smote\")\n",
        "100 * text4.count('a') / len(text4)"
      ],
      "metadata": {
        "id": "DVYPxtfbJ76i"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Definition of some functions helpful to measure the lexical diversity and its percentage in a text."
      ],
      "metadata": {
        "id": "78WAm731OH3x"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def lexical_diversity(text):\n",
        "  return len(set(text)) / len(text)\n",
        "\n",
        "def percentage(count, total):\n",
        "  return 100 * count / total"
      ],
      "metadata": {
        "id": "JGiuJ7EyN1KL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Frequency Distribtuions\n",
        "NLTK provide a way to see the frequency distribution, telling us the frequency of each vocabulary item in the text with the possibilty to plot it."
      ],
      "metadata": {
        "id": "k_VjURSCR_l6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "fdist1 = FreqDist(text1)\n",
        "print(fdist1)\n",
        "fdist1.most_common(50)\n",
        "fdist1['whale']\n",
        "fdist1.plot(50, cumulative=True)"
      ],
      "metadata": {
        "id": "xHcCBZ2KSUqX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "If your goal is to identify the meaning of a particular text and the frequent words don't help up we can check for the ones that occure once only using the **hapaxes**."
      ],
      "metadata": {
        "id": "t6l0Rdm5T62M"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "fdist1.hapaxes()"
      ],
      "metadata": {
        "id": "02aenBQ-USlJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "If the purpose is to take a look for some words that comply a particular property it's possible use some notation from *set theory*:\n",
        "\n",
        "\n",
        "$\\{ w \\mid w \\in V \\ \\&\\ P(w) \\}$\n",
        "\n",
        "\n",
        "in Python the corresponding expression is given by:\n",
        "\n",
        "\n",
        "[w for w in V if p(w)]"
      ],
      "metadata": {
        "id": "ga-M5ID0U46Q"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "V = set(text1)\n",
        "long_words = [w for w in V if len(w) > 15]\n",
        "sorted(long_words)"
      ],
      "metadata": {
        "id": "BHMRUGpDW0BM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Following our goal (find words that characterize a text) the next task is to clean \"the noise\" given by hapaxes and *antiphilophists* (infrequent long words)."
      ],
      "metadata": {
        "id": "WN6f23tsXugV"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "fdist5 = FreqDist(text5)\n",
        "sorted(w for w in set(text5) if len(w) > 7 and fdist5[w] > 7)"
      ],
      "metadata": {
        "id": "UEvEIGv8YNiA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Collocations and Bigrams\n",
        "A **collocation** is a sequence of words that occur together unusually often. To get a handle on collocations, we start off by extracting from a text a list of word pairs, also known as **bigrams**."
      ],
      "metadata": {
        "id": "YU9Ak-pfavVd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "list(bigrams([\"more\", \"is\", \"said\", \"than\", \"done\"]))"
      ],
      "metadata": {
        "id": "UA8dqFFEcCDL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "According its definition, collocations are essentially just frequent bigrams. It's possible to find bigrams that occur more often based on frequency of the individual words."
      ],
      "metadata": {
        "id": "PL6cE2cFdmro"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "text4.collocations()\n",
        "text8.collocations()"
      ],
      "metadata": {
        "id": "0E8UHdUHd5Sb"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Counting Other Things\n",
        "To look at the distribution of word lengths in a text we start by deriving a list of the lengths of words, then counts the number of times each of these occurs. To see how frequent the different lengths of word are is possible do it using the following code."
      ],
      "metadata": {
        "id": "EIxFhe3feHAs"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "[len(w) for w in text1]\n",
        "fdist = FreqDist(len(w) for w in text1)\n",
        "print(fdist)\n",
        "fdist\n",
        "\n",
        "fdist.most_common()\n",
        "fdist.max()\n",
        "fdist[3]\n",
        "fdist.freq(3)"
      ],
      "metadata": {
        "id": "R6_rSo1Eiv_y"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "From this measure we can notice that the most frequent word length is 3 and appear for roughly 50000 (20%) of the words making up the book."
      ],
      "metadata": {
        "id": "p2bv6XYijRm2"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "SZQYvCUHmUN_"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}