AmItheAssholeAnalysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## r/AmItheAsshole Lanugage Analysis\n",
    "\n",
    "### Introduction\n",
    "\n",
    "[r/AmItheAsshole](https://www.reddit.com/r/AmItheAsshole/) is a subreddit where people will post a story about some conflict they've had and it is up to the community to judge who the \"asshole\" is in the situation. The page description words this very eloquently:\n",
    "\n",
    "\"A catharsis for the frustrated moral philosopher in all of us, and a place to finally find out if you were wrong in an argument that's been bothering you. Tell us about any non-violent conflict you have experienced; give us both sides of the story, and find out if you're right, or you're the asshole.\" ([r/AmItheAsshole](https://www.reddit.com/r/AmItheAsshole/))\n",
    "\n",
    "The goal of this project if there are any notable lexical features between the langauge of the \"assholes\" and those who are not.\n",
    "\n",
    "One thing that I would like to note is that this is not a corpus directly translated from speech so the language of the mosts might have been thoughfully worded and might not be representative of ones's natural speech. Still I believe that the judgments voted on by the reddit community can provide insight on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scraping from Reddit\n",
    "#### PRAW\n",
    "[The Python Reddit API Wrapper (PRAW)](https://praw.readthedocs.io/en/v4.1.0/index.html) is an API for anything you can do in Reddit. In this project it is used to scrape posts from the r/AmItheAsshole subreddit.\n",
    "\n",
    "#### The r/AmItheAsshole Archives\n",
    "The subreddit archives the posts into 4 different categories related to this project. There are 5 judgements that one can decide on, however the last one that is not archived 'INFO' (Not enough info) does not help with this investigation. The different judgements are described in the function below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Helper functions for scraping from reddit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import glob\n",
    "import yaml\n",
    "import praw\n",
    "import pandas as pd\n",
    "from tqdm import tqdm_notebook\n",
    "\n",
    "\n",
    "\n",
    "# Initialize reddit API\n",
    "reddit = praw.Reddit('scraper', user_agent='corpus ling project')\n",
    "reddit.read_only = True\n",
    "\n",
    "# Abbreviations for the archive queries\n",
    "archive_names = { 'YTA' : 'Asshole', 'NTA' : 'Not the A-hole', 'ESH' : 'Everyone Sucks',\n",
    "        'NAH' : 'No A-holes here'}\n",
    "\n",
    "# Pandas dataframe containing all of the posts.\n",
    "corpus = pd.DataFrame()\n",
    "\n",
    "def get_archive(judgment):\n",
    "    '''\n",
    "    Get the posts from a single archive of r/AmItheAsshole\n",
    "    Archive judgement options are:\n",
    "        'YTA' -- You're the Asshole (& the other party is not)\n",
    "        'NTA' -- You're Not the A-hole (& the other party is)\n",
    "        'ESH' -- Everyone Sucks Here\n",
    "        'NAH' -- No A-holes here\n",
    "    '''\n",
    "    return reddit.subreddit('AmItheAsshole').search(\n",
    "            'flair_name:\"' + archive_names[judgment] + '\"', limit=None)\n",
    "\n",
    "def load_archives(download_local = True):\n",
    "    '''\n",
    "    Grabs each of the archives and puts them in the corpus dictionary.\n",
    "    The download_local flag determines if you want to download the text\n",
    "    as well in the current directory.\n",
    "    '''\n",
    "    global corpus \n",
    "    corpus = pd.DataFrame()\n",
    "    for judgement in tqdm_notebook(archive_names.keys()):\n",
    "        posts = get_archive(judgement)\n",
    "        row_df = pd.io.json.json_normalize(\n",
    "                map(lambda post : {'archive' : judgement, **vars(post)}, posts))\n",
    "        # store only the id, title, archive, rawtext, and url\n",
    "        corpus = corpus.append(row_df[['id', 'title', 'archive', 'selftext', 'url']],\n",
    "                               ignore_index = True, sort=True)        \n",
    "        \n",
    "        # Download the text locally\n",
    "        if (download_local):\n",
    "            if not os.path.exists(judgement):\n",
    "                # separate archives into different directories.\n",
    "                os.mkdir(judgement)\n",
    "            for index, post in (corpus.loc[corpus['archive'] == judgement]).iterrows():\n",
    "                # Save each file in their respective directories with their id as the filename\n",
    "                with open(judgement + '/' + post.id + '.yaml', 'w') as file:\n",
    "                    documents = yaml.dump(post.to_dict(), file)\n",
    "\n",
    "def local_load_archives():\n",
    "    '''\n",
    "    Loads the archived posts from yaml files in the current directory.\n",
    "    '''\n",
    "    global corpus \n",
    "    corpus = pd.DataFrame()\n",
    "    for judgement in tqdm_notebook(archive_names.keys()):\n",
    "        for filename in glob.glob(judgement + '/*.yaml'):\n",
    "            with open(filename) as file:\n",
    "                corpus = corpus.append(pd.io.json.json_normalize(\n",
    "                    yaml.load(file, Loader=yaml.FullLoader)),\n",
    "                                       ignore_index = True, sort=True)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Several options for loading the corpus of posts:\n",
    "To keep things consistent, please use the `local_load_archives()` option becuase posts might be added over time and so the anaysis of this corpus may change as new posts are added.\n",
    "\n",
    "Note: `load_archives()` will not work unless you have a `praw.ini` file in the current directory. This file contains the necesary account information for using the Reddit API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this if you just want to use the posts that are downloaded locally.\n",
    "local_load_archives()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this if you want the most recent set of posts but without downloading them locally.\n",
    "load_archives(False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this if you want to load and update the local files with the current archives.\n",
    "load_archives()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The corpus\n",
    "The corpus of posts is represented as a datarame whos most important columns are the archive column, and the text. The following code prints out summaries of all of the archives with the counts of posts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      " Summary of all of the posts in the \"Asshole\" archive:\n",
      "       archive      id                                           selftext  \\\n",
      "count      238     238                                                238   \n",
      "unique       1     238                                                238   \n",
      "top        YTA  eaghx8  My sisters boyfriend came over for thanksgivin...   \n",
      "freq       238       1                                                  1   \n",
      "\n",
      "                                                    title  \\\n",
      "count                                                 238   \n",
      "unique                                                238   \n",
      "top     AITA that I didn't give up my bus seat for a p...   \n",
      "freq                                                    1   \n",
      "\n",
      "                                                      url  \n",
      "count                                                 238  \n",
      "unique                                                238  \n",
      "top     https://www.reddit.com/r/AmItheAsshole/comment...  \n",
      "freq                                                    1  \n",
      "\n",
      "\n",
      " Summary of all of the posts in the \"Not the A-hole\" archive:\n",
      "       archive      id                                           selftext  \\\n",
      "count      260     260                                                260   \n",
      "unique       1     260                                                260   \n",
      "top        NTA  e4pfqz  My sister recently went through a messy break ...   \n",
      "freq       260       1                                                  1   \n",
      "\n",
      "                                                    title  \\\n",
      "count                                                 260   \n",
      "unique                                                260   \n",
      "top     AITA for refusing to meet my half sister who w...   \n",
      "freq                                                    1   \n",
      "\n",
      "                                                      url  \n",
      "count                                                 260  \n",
      "unique                                                260  \n",
      "top     https://www.reddit.com/r/AmItheAsshole/comment...  \n",
      "freq                                                    1  \n",
      "\n",
      "\n",
      " Summary of all of the posts in the \"Everyone Sucks\" archive:\n",
      "       archive      id                                           selftext  \\\n",
      "count      245     245                                                245   \n",
      "unique       1     245                                                245   \n",
      "top        ESH  dj0z9h  Throw away account. \\nI know I seem like the a...   \n",
      "freq       245       1                                                  1   \n",
      "\n",
      "                                                    title  \\\n",
      "count                                                 245   \n",
      "unique                                                245   \n",
      "top     AITA for having my niece stay the weekend with...   \n",
      "freq                                                    1   \n",
      "\n",
      "                                                      url  \n",
      "count                                                 245  \n",
      "unique                                                245  \n",
      "top     https://www.reddit.com/r/AmItheAsshole/comment...  \n",
      "freq                                                    1  \n",
      "\n",
      "\n",
      " Summary of all of the posts in the \"No A-holes here\" archive:\n",
      "       archive      id                                           selftext  \\\n",
      "count      261     261                                                261   \n",
      "unique       1     261                                                261   \n",
      "top        NAH  dep6h9  So, my gf bought me a 23 and me test for my la...   \n",
      "freq       261       1                                                  1   \n",
      "\n",
      "                                                    title  \\\n",
      "count                                                 261   \n",
      "unique                                                261   \n",
      "top     AITA for telling my fiance that I want to be a...   \n",
      "freq                                                    1   \n",
      "\n",
      "                                                      url  \n",
      "count                                                 261  \n",
      "unique                                                261  \n",
      "top     https://www.reddit.com/r/AmItheAsshole/comment...  \n",
      "freq                                                    1  \n"
     ]
    }
   ],
   "source": [
    "for judgement in archive_names.keys():\n",
    "    print(\"\\n\\n Summary of all of the posts in the \\\"\" + archive_names[judgement] + \"\\\" archive:\")\n",
    "    print(corpus.loc[corpus['archive'] == judgement].describe())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Feel free to take a break and read a random post:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Been married seven years. I'm 35M, she's 34F. We have one daughter 5F and one son 9moM. For more context about our family: I make a lot of money. My wife is unemployed and has been unemployed since we met. The house is mine, the two cars are mine, etc. I mean, they're both of ours, but I purchased them for the family.\n",
      "\n",
      "My wife decided to spring onto me two weeks ago that she wants to explore a polyromantic relationship. She has no one in mind but is excited about the potential and asks if I'm up for it. My answer is a resounding no because I agreed to monogamy when we married. We actively discussed this beforehand, too, and I thought we were on the same page.\n",
      "\n",
      "This is where I may be the asshole: my wife has zero bargaining power here. To put it very bluntly, she doesn't make enough money to make this kind of suggestion or demand. She has said me not accepting her polyromanticism is putting our relationship into jeopardy and while I may lose a lot in a divorce or separation, she's 34 with no degree, no job prospects, no ambition to be honest. Her life would be hard, even with alimony. Mine would not, nor would our children.\n",
      "\n",
      "I won't budge and it seems like she won't budge. No amount of discussion has resolved this, we're both set in our ways. It looks like separation is on the horizon.\n",
      "\n",
      "All it would take is for me to accept things and our family remains whole, but I won't. Our family will break apart because we can't come to an agreement. Our children will go to-and-fro between two homes, maybe many stepparents. I feel for our children but I have to stick by my own principles, right?\n",
      "\n",
      "AITA?\n"
     ]
    }
   ],
   "source": [
    "post = corpus.sample(n=1)\n",
    "print(post.selftext.tolist()[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Can you guess how the community judged this submission?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Everyone Sucks\n"
     ]
    }
   ],
   "source": [
    "print(archive_names[post.archive.tolist()[0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analyzing the Corpus\n",
    "#### NLTK\n",
    "[Natural Language Toolkit (NLTK)](https://www.nltk.org/) is a python library with a lot of Natural Language processing tools.\n",
    "\n",
    "First we need to import nltk and download some tools:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "nltk.download('punkt')\n",
    "nltk.download('stopwords')\n",
    "nltk.download('averaged_perceptron_tagger')\n",
    "nltk.download('vader_lexicon')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some helper functions for using the nltk library on the corpus dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk import word_tokenize, sent_tokenize, Text, FreqDist, pos_tag\n",
    "\n",
    "def corpus_tokenize():\n",
    "    '''\n",
    "    Converts the text of each post to a list of tokens and places\n",
    "    them in a list in a column within the corpus dataframe.\n",
    "    '''\n",
    "    global corpus\n",
    "    corpus['tokens'] = list(map(lambda text : word_tokenize(text.lower()), corpus['selftext']))\n",
    "    # Get rid of punctuation\n",
    "    tokenizer = nltk.tokenize.RegexpTokenizer(r'\\w+')\n",
    "    corpus['tokens_no_punct'] = list(map(lambda text : tokenizer.tokenize(text.lower()), corpus['selftext']))\n",
    "    # Tokenize on sentences.\n",
    "    corpus['sent_tokens'] = list(map(sent_tokenize, corpus['selftext']))\n",
    "    \n",
    "def corpus_pos_tags():\n",
    "    '''\n",
    "    POS tag the whole corpus and put it into a new column. Only\n",
    "    run after corpus_tokenize().\n",
    "    '''\n",
    "    global corpus\n",
    "    corpus['pos_tags'] = list(map(pos_tag, corpus['tokens']))\n",
    "\n",
    "    \n",
    "def get_archive_texts(archives, no_punctuation = False):\n",
    "    '''\n",
    "    Gets the texts of every post of the archives listed in archives\n",
    "    and returns it as one list of words.\n",
    "    '''\n",
    "    global corpus\n",
    "    texts = []\n",
    "    for archive in archives:\n",
    "        for post_tokens in corpus.loc[corpus['archive'] == archive]['tokens_no_punct' if no_punctuation else 'tokens'].to_list():\n",
    "            texts += post_tokens\n",
    "    return texts\n",
    "\n",
    "def get_archive_texts_sentences(archives):\n",
    "    '''\n",
    "    Gets the texts of every post of the archives listed in archives\n",
    "    and returns it as one list of sentences.\n",
    "    '''\n",
    "    global corpus\n",
    "    texts = []\n",
    "    for archive in archives:\n",
    "        for sent_tokens in corpus.loc[corpus['archive'] == archive]['sent_tokens'].to_list():\n",
    "            texts += sent_tokens\n",
    "    return texts\n",
    "\n",
    "def get_archive_texts_pos_tags(archives):\n",
    "    '''\n",
    "    Gets the pos tags of every post of the archives listed in archives\n",
    "    and returns it as one list of pos tags.\n",
    "    '''\n",
    "    global corpus\n",
    "    texts = []\n",
    "    for archive in archives:\n",
    "        for sent_tokens in corpus.loc[corpus['archive'] == archive]['pos_tags'].to_list():\n",
    "            texts += sent_tokens\n",
    "    return texts\n",
    "\n",
    "def get_freq_dist(archives, no_stopwords = False):\n",
    "    '''\n",
    "    Get a FreqDist for the given archive names There is also the option to remove stopwords.\n",
    "    '''\n",
    "    text = get_archive_texts(archives, no_punctuation=True)\n",
    "    if no_stopwords:\n",
    "        # filter out stopwords\n",
    "        stopwords = nltk.corpus.stopwords.words('english')\n",
    "        text = list(filter(lambda word : not word in stopwords, text))\n",
    "    return FreqDist(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Preprocessing the Corpus\n",
    "Creates three new columns in the corpus dataframe, 'tokens', 'tokens_no_punct', and 'sent_tokens'. One with just the raw output of the nltk tokenize function, the next without punctiation, and finally the text separated by sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>archive</th>\n",
       "      <th>id</th>\n",
       "      <th>selftext</th>\n",
       "      <th>title</th>\n",
       "      <th>url</th>\n",
       "      <th>tokens</th>\n",
       "      <th>tokens_no_punct</th>\n",
       "      <th>sent_tokens</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>YTA</td>\n",
       "      <td>e8l9h6</td>\n",
       "      <td>My husband (30M) is defending his PhD in 1 wee...</td>\n",
       "      <td>WIBTA if I didn't go to my husband's PhD defense</td>\n",
       "      <td>https://www.reddit.com/r/AmItheAsshole/comment...</td>\n",
       "      <td>[my, husband, (, 30m, ), is, defending, his, p...</td>\n",
       "      <td>[my, husband, 30m, is, defending, his, phd, in...</td>\n",
       "      <td>[My husband (30M) is defending his PhD in 1 we...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>YTA</td>\n",
       "      <td>ec6a0d</td>\n",
       "      <td>Okay, to establish this, my ex and I have been...</td>\n",
       "      <td>AITA for not offering to help my ex pay for di...</td>\n",
       "      <td>https://www.reddit.com/r/AmItheAsshole/comment...</td>\n",
       "      <td>[okay, ,, to, establish, this, ,, my, ex, and,...</td>\n",
       "      <td>[okay, to, establish, this, my, ex, and, i, ha...</td>\n",
       "      <td>[Okay, to establish this, my ex and I have bee...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>YTA</td>\n",
       "      <td>duxns4</td>\n",
       "      <td>So Thursday, I went to a party one of my frien...</td>\n",
       "      <td>AITA for being livid at my roommate for callin...</td>\n",
       "      <td>https://www.reddit.com/r/AmItheAsshole/comment...</td>\n",
       "      <td>[so, thursday, ,, i, went, to, a, party, one, ...</td>\n",
       "      <td>[so, thursday, i, went, to, a, party, one, of,...</td>\n",
       "      <td>[So Thursday, I went to a party one of my frie...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>YTA</td>\n",
       "      <td>e617k1</td>\n",
       "      <td>I wanted to do something nice for my mom for h...</td>\n",
       "      <td>AITA for not painting my step father in a fami...</td>\n",
       "      <td>https://www.reddit.com/r/AmItheAsshole/comment...</td>\n",
       "      <td>[i, wanted, to, do, something, nice, for, my, ...</td>\n",
       "      <td>[i, wanted, to, do, something, nice, for, my, ...</td>\n",
       "      <td>[I wanted to do something nice for my mom for ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>YTA</td>\n",
       "      <td>e8tpjf</td>\n",
       "      <td>Edit: threw OUT* a diaper beside my head\\n\\nTh...</td>\n",
       "      <td>AITA for being mad someone threw a diaper next...</td>\n",
       "      <td>https://www.reddit.com/r/AmItheAsshole/comment...</td>\n",
       "      <td>[edit, :, threw, out*, a, diaper, beside, my, ...</td>\n",
       "      <td>[edit, threw, out, a, diaper, beside, my, head...</td>\n",
       "      <td>[Edit: threw OUT* a diaper beside my head\\n\\nT...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  archive      id                                           selftext  \\\n",
       "0     YTA  e8l9h6  My husband (30M) is defending his PhD in 1 wee...   \n",
       "1     YTA  ec6a0d  Okay, to establish this, my ex and I have been...   \n",
       "2     YTA  duxns4  So Thursday, I went to a party one of my frien...   \n",
       "3     YTA  e617k1  I wanted to do something nice for my mom for h...   \n",
       "4     YTA  e8tpjf  Edit: threw OUT* a diaper beside my head\\n\\nTh...   \n",
       "\n",
       "                                               title  \\\n",
       "0   WIBTA if I didn't go to my husband's PhD defense   \n",
       "1  AITA for not offering to help my ex pay for di...   \n",
       "2  AITA for being livid at my roommate for callin...   \n",
       "3  AITA for not painting my step father in a fami...   \n",
       "4  AITA for being mad someone threw a diaper next...   \n",
       "\n",
       "                                                 url  \\\n",
       "0  https://www.reddit.com/r/AmItheAsshole/comment...   \n",
       "1  https://www.reddit.com/r/AmItheAsshole/comment...   \n",
       "2  https://www.reddit.com/r/AmItheAsshole/comment...   \n",
       "3  https://www.reddit.com/r/AmItheAsshole/comment...   \n",
       "4  https://www.reddit.com/r/AmItheAsshole/comment...   \n",
       "\n",
       "                                              tokens  \\\n",
       "0  [my, husband, (, 30m, ), is, defending, his, p...   \n",
       "1  [okay, ,, to, establish, this, ,, my, ex, and,...   \n",
       "2  [so, thursday, ,, i, went, to, a, party, one, ...   \n",
       "3  [i, wanted, to, do, something, nice, for, my, ...   \n",
       "4  [edit, :, threw, out*, a, diaper, beside, my, ...   \n",
       "\n",
       "                                     tokens_no_punct  \\\n",
       "0  [my, husband, 30m, is, defending, his, phd, in...   \n",
       "1  [okay, to, establish, this, my, ex, and, i, ha...   \n",
       "2  [so, thursday, i, went, to, a, party, one, of,...   \n",
       "3  [i, wanted, to, do, something, nice, for, my, ...   \n",
       "4  [edit, threw, out, a, diaper, beside, my, head...   \n",
       "\n",
       "                                         sent_tokens  \n",
       "0  [My husband (30M) is defending his PhD in 1 we...  \n",
       "1  [Okay, to establish this, my ex and I have bee...  \n",
       "2  [So Thursday, I went to a party one of my frie...  \n",
       "3  [I wanted to do something nice for my mom for ...  \n",
       "4  [Edit: threw OUT* a diaper beside my head\\n\\nT...  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus_tokenize()\n",
    "corpus.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Word Frequencies\n",
    "Since these posts are written in the first person and talk about personal experiences, a lot of the most frequent words are pronouns, therefore the following frequency counts remove stopwords. This can be changed by replacing `True` with `False` in the no_stopwords field."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Word</th>\n",
       "      <th>Frequency</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>like</td>\n",
       "      <td>1455</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>said</td>\n",
       "      <td>1388</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>would</td>\n",
       "      <td>1175</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>told</td>\n",
       "      <td>1156</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>time</td>\n",
       "      <td>1065</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>get</td>\n",
       "      <td>1006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>one</td>\n",
       "      <td>955</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>got</td>\n",
       "      <td>908</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>want</td>\n",
       "      <td>895</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>really</td>\n",
       "      <td>877</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>know</td>\n",
       "      <td>807</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>go</td>\n",
       "      <td>737</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>people</td>\n",
       "      <td>704</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>wife</td>\n",
       "      <td>702</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>think</td>\n",
       "      <td>667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>family</td>\n",
       "      <td>654</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>going</td>\n",
       "      <td>649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>even</td>\n",
       "      <td>632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>us</td>\n",
       "      <td>632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>also</td>\n",
       "      <td>625</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Word  Frequency\n",
       "0     like       1455\n",
       "1     said       1388\n",
       "2    would       1175\n",
       "3     told       1156\n",
       "4     time       1065\n",
       "5      get       1006\n",
       "6      one        955\n",
       "7      got        908\n",
       "8     want        895\n",
       "9   really        877\n",
       "10    know        807\n",
       "11      go        737\n",
       "12  people        704\n",
       "13    wife        702\n",
       "14   think        667\n",
       "15  family        654\n",
       "16   going        649\n",
       "17    even        632\n",
       "18      us        632\n",
       "19    also        625"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(get_freq_dist(archive_names.keys(), no_stopwords=True).most_common(20), columns=['Word', 'Frequency'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word_a_hole</th>\n",
       "      <th>frequency_a_hole</th>\n",
       "      <th>word</th>\n",
       "      <th>frequency</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>like</td>\n",
       "      <td>715</td>\n",
       "      <td>like</td>\n",
       "      <td>740</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>said</td>\n",
       "      <td>690</td>\n",
       "      <td>said</td>\n",
       "      <td>698</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>told</td>\n",
       "      <td>541</td>\n",
       "      <td>would</td>\n",
       "      <td>654</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>time</td>\n",
       "      <td>539</td>\n",
       "      <td>told</td>\n",
       "      <td>615</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>would</td>\n",
       "      <td>521</td>\n",
       "      <td>time</td>\n",
       "      <td>526</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>get</td>\n",
       "      <td>480</td>\n",
       "      <td>get</td>\n",
       "      <td>526</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>one</td>\n",
       "      <td>447</td>\n",
       "      <td>want</td>\n",
       "      <td>509</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>got</td>\n",
       "      <td>419</td>\n",
       "      <td>one</td>\n",
       "      <td>508</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>know</td>\n",
       "      <td>387</td>\n",
       "      <td>really</td>\n",
       "      <td>495</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>want</td>\n",
       "      <td>386</td>\n",
       "      <td>got</td>\n",
       "      <td>489</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>really</td>\n",
       "      <td>382</td>\n",
       "      <td>know</td>\n",
       "      <td>420</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>go</td>\n",
       "      <td>376</td>\n",
       "      <td>people</td>\n",
       "      <td>380</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>wife</td>\n",
       "      <td>355</td>\n",
       "      <td>family</td>\n",
       "      <td>374</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>even</td>\n",
       "      <td>346</td>\n",
       "      <td>go</td>\n",
       "      <td>361</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>us</td>\n",
       "      <td>333</td>\n",
       "      <td>also</td>\n",
       "      <td>351</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>people</td>\n",
       "      <td>324</td>\n",
       "      <td>going</td>\n",
       "      <td>347</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>think</td>\n",
       "      <td>322</td>\n",
       "      <td>wife</td>\n",
       "      <td>347</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>going</td>\n",
       "      <td>302</td>\n",
       "      <td>think</td>\n",
       "      <td>345</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>work</td>\n",
       "      <td>294</td>\n",
       "      <td>feel</td>\n",
       "      <td>342</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>year</td>\n",
       "      <td>290</td>\n",
       "      <td>years</td>\n",
       "      <td>331</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>back</td>\n",
       "      <td>287</td>\n",
       "      <td>asked</td>\n",
       "      <td>324</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>asked</td>\n",
       "      <td>285</td>\n",
       "      <td>mom</td>\n",
       "      <td>323</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>family</td>\n",
       "      <td>280</td>\n",
       "      <td>back</td>\n",
       "      <td>313</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>also</td>\n",
       "      <td>274</td>\n",
       "      <td>home</td>\n",
       "      <td>307</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>could</td>\n",
       "      <td>273</td>\n",
       "      <td>us</td>\n",
       "      <td>299</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>since</td>\n",
       "      <td>272</td>\n",
       "      <td>since</td>\n",
       "      <td>295</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>home</td>\n",
       "      <td>270</td>\n",
       "      <td>work</td>\n",
       "      <td>295</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>years</td>\n",
       "      <td>268</td>\n",
       "      <td>year</td>\n",
       "      <td>295</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>make</td>\n",
       "      <td>260</td>\n",
       "      <td>even</td>\n",
       "      <td>286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>kids</td>\n",
       "      <td>259</td>\n",
       "      <td>make</td>\n",
       "      <td>286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>daughter</td>\n",
       "      <td>249</td>\n",
       "      <td>much</td>\n",
       "      <td>279</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>feel</td>\n",
       "      <td>244</td>\n",
       "      <td>kids</td>\n",
       "      <td>276</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>went</td>\n",
       "      <td>243</td>\n",
       "      <td>could</td>\n",
       "      <td>274</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>day</td>\n",
       "      <td>242</td>\n",
       "      <td>things</td>\n",
       "      <td>261</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>never</td>\n",
       "      <td>242</td>\n",
       "      <td>aita</td>\n",
       "      <td>260</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>aita</td>\n",
       "      <td>237</td>\n",
       "      <td>dad</td>\n",
       "      <td>259</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>house</td>\n",
       "      <td>235</td>\n",
       "      <td>parents</td>\n",
       "      <td>257</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>say</td>\n",
       "      <td>234</td>\n",
       "      <td>first</td>\n",
       "      <td>256</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>friends</td>\n",
       "      <td>231</td>\n",
       "      <td>day</td>\n",
       "      <td>247</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>edit</td>\n",
       "      <td>230</td>\n",
       "      <td>well</td>\n",
       "      <td>243</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>parents</td>\n",
       "      <td>228</td>\n",
       "      <td>never</td>\n",
       "      <td>242</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>husband</td>\n",
       "      <td>225</td>\n",
       "      <td>still</td>\n",
       "      <td>238</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>things</td>\n",
       "      <td>224</td>\n",
       "      <td>something</td>\n",
       "      <td>231</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>asshole</td>\n",
       "      <td>214</td>\n",
       "      <td>2</td>\n",
       "      <td>229</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>something</td>\n",
       "      <td>209</td>\n",
       "      <td>went</td>\n",
       "      <td>228</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>well</td>\n",
       "      <td>208</td>\n",
       "      <td>always</td>\n",
       "      <td>225</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>much</td>\n",
       "      <td>206</td>\n",
       "      <td>edit</td>\n",
       "      <td>225</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>still</td>\n",
       "      <td>204</td>\n",
       "      <td>friends</td>\n",
       "      <td>222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>school</td>\n",
       "      <td>201</td>\n",
       "      <td>lot</td>\n",
       "      <td>221</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>first</td>\n",
       "      <td>195</td>\n",
       "      <td>see</td>\n",
       "      <td>220</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   word_a_hole  frequency_a_hole       word  frequency\n",
       "0         like               715       like        740\n",
       "1         said               690       said        698\n",
       "2         told               541      would        654\n",
       "3         time               539       told        615\n",
       "4        would               521       time        526\n",
       "5          get               480        get        526\n",
       "6          one               447       want        509\n",
       "7          got               419        one        508\n",
       "8         know               387     really        495\n",
       "9         want               386        got        489\n",
       "10      really               382       know        420\n",
       "11          go               376     people        380\n",
       "12        wife               355     family        374\n",
       "13        even               346         go        361\n",
       "14          us               333       also        351\n",
       "15      people               324      going        347\n",
       "16       think               322       wife        347\n",
       "17       going               302      think        345\n",
       "18        work               294       feel        342\n",
       "19        year               290      years        331\n",
       "20        back               287      asked        324\n",
       "21       asked               285        mom        323\n",
       "22      family               280       back        313\n",
       "23        also               274       home        307\n",
       "24       could               273         us        299\n",
       "25       since               272      since        295\n",
       "26        home               270       work        295\n",
       "27       years               268       year        295\n",
       "28        make               260       even        286\n",
       "29        kids               259       make        286\n",
       "30    daughter               249       much        279\n",
       "31        feel               244       kids        276\n",
       "32        went               243      could        274\n",
       "33         day               242     things        261\n",
       "34       never               242       aita        260\n",
       "35        aita               237        dad        259\n",
       "36       house               235    parents        257\n",
       "37         say               234      first        256\n",
       "38     friends               231        day        247\n",
       "39        edit               230       well        243\n",
       "40     parents               228      never        242\n",
       "41     husband               225      still        238\n",
       "42      things               224  something        231\n",
       "43     asshole               214          2        229\n",
       "44   something               209       went        228\n",
       "45        well               208     always        225\n",
       "46        much               206       edit        225\n",
       "47       still               204    friends        222\n",
       "48      school               201        lot        221\n",
       "49       first               195        see        220"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_most_common = 50\n",
    "\n",
    "pd.DataFrame(get_freq_dist(['YTA', 'ESH'], no_stopwords=True).most_common(n_most_common),\n",
    "             columns=['word', 'frequency']).join(\n",
    "    pd.DataFrame(get_freq_dist(['NTA', 'NAH'], no_stopwords=True).most_common(n_most_common),\n",
    "                 columns=['word', 'frequency']), lsuffix=\"_a_hole\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### POS tags\n",
    "First we need to calculate all of the POS tags. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus_pos_tags()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How do the frequencies of the different tags compare?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a_hole_pos_count</th>\n",
       "      <th>pos_count</th>\n",
       "      <th>total</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>NN</th>\n",
       "      <td>31312</td>\n",
       "      <td>34864</td>\n",
       "      <td>66176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>IN</th>\n",
       "      <td>19320</td>\n",
       "      <td>21324</td>\n",
       "      <td>40644</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PRP</th>\n",
       "      <td>14628</td>\n",
       "      <td>15535</td>\n",
       "      <td>30163</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DT</th>\n",
       "      <td>13611</td>\n",
       "      <td>15230</td>\n",
       "      <td>28841</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>JJ</th>\n",
       "      <td>13426</td>\n",
       "      <td>15264</td>\n",
       "      <td>28690</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>RB</th>\n",
       "      <td>13200</td>\n",
       "      <td>14884</td>\n",
       "      <td>28084</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VBD</th>\n",
       "      <td>10780</td>\n",
       "      <td>11512</td>\n",
       "      <td>22292</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VB</th>\n",
       "      <td>9922</td>\n",
       "      <td>11190</td>\n",
       "      <td>21112</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>.</th>\n",
       "      <td>9743</td>\n",
       "      <td>10718</td>\n",
       "      <td>20461</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CC</th>\n",
       "      <td>8371</td>\n",
       "      <td>9271</td>\n",
       "      <td>17642</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PRP$</th>\n",
       "      <td>7218</td>\n",
       "      <td>8246</td>\n",
       "      <td>15464</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NNS</th>\n",
       "      <td>6829</td>\n",
       "      <td>7930</td>\n",
       "      <td>14759</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VBP</th>\n",
       "      <td>6881</td>\n",
       "      <td>7768</td>\n",
       "      <td>14649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>,</th>\n",
       "      <td>5823</td>\n",
       "      <td>7155</td>\n",
       "      <td>12978</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TO</th>\n",
       "      <td>5699</td>\n",
       "      <td>6392</td>\n",
       "      <td>12091</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VBZ</th>\n",
       "      <td>5375</td>\n",
       "      <td>5827</td>\n",
       "      <td>11202</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VBG</th>\n",
       "      <td>4809</td>\n",
       "      <td>5421</td>\n",
       "      <td>10230</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>VBN</th>\n",
       "      <td>3487</td>\n",
       "      <td>3841</td>\n",
       "      <td>7328</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MD</th>\n",
       "      <td>2383</td>\n",
       "      <td>2706</td>\n",
       "      <td>5089</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CD</th>\n",
       "      <td>2263</td>\n",
       "      <td>2499</td>\n",
       "      <td>4762</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>RP</th>\n",
       "      <td>1459</td>\n",
       "      <td>1482</td>\n",
       "      <td>2941</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>WRB</th>\n",
       "      <td>1337</td>\n",
       "      <td>1511</td>\n",
       "      <td>2848</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>:</th>\n",
       "      <td>813</td>\n",
       "      <td>1054</td>\n",
       "      <td>1867</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>)</th>\n",
       "      <td>798</td>\n",
       "      <td>934</td>\n",
       "      <td>1732</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>WP</th>\n",
       "      <td>829</td>\n",
       "      <td>875</td>\n",
       "      <td>1704</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>(</th>\n",
       "      <td>782</td>\n",
       "      <td>886</td>\n",
       "      <td>1668</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NNP</th>\n",
       "      <td>792</td>\n",
       "      <td>831</td>\n",
       "      <td>1623</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>WDT</th>\n",
       "      <td>540</td>\n",
       "      <td>602</td>\n",
       "      <td>1142</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>JJR</th>\n",
       "      <td>421</td>\n",
       "      <td>536</td>\n",
       "      <td>957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>''</th>\n",
       "      <td>465</td>\n",
       "      <td>483</td>\n",
       "      <td>948</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>``</th>\n",
       "      <td>423</td>\n",
       "      <td>446</td>\n",
       "      <td>869</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>JJS</th>\n",
       "      <td>327</td>\n",
       "      <td>337</td>\n",
       "      <td>664</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>POS</th>\n",
       "      <td>260</td>\n",
       "      <td>309</td>\n",
       "      <td>569</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EX</th>\n",
       "      <td>273</td>\n",
       "      <td>287</td>\n",
       "      <td>560</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>RBR</th>\n",
       "      <td>247</td>\n",
       "      <td>265</td>\n",
       "      <td>512</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PDT</th>\n",
       "      <td>172</td>\n",
       "      <td>221</td>\n",
       "      <td>393</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>$</th>\n",
       "      <td>144</td>\n",
       "      <td>142</td>\n",
       "      <td>286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>FW</th>\n",
       "      <td>82</td>\n",
       "      <td>117</td>\n",
       "      <td>199</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>#</th>\n",
       "      <td>70</td>\n",
       "      <td>96</td>\n",
       "      <td>166</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>UH</th>\n",
       "      <td>51</td>\n",
       "      <td>74</td>\n",
       "      <td>125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>RBS</th>\n",
       "      <td>52</td>\n",
       "      <td>56</td>\n",
       "      <td>108</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SYM</th>\n",
       "      <td>5</td>\n",
       "      <td>8</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>WP$</th>\n",
       "      <td>8</td>\n",
       "      <td>3</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      a_hole_pos_count  pos_count  total\n",
       "NN               31312      34864  66176\n",
       "IN               19320      21324  40644\n",
       "PRP              14628      15535  30163\n",
       "DT               13611      15230  28841\n",
       "JJ               13426      15264  28690\n",
       "RB               13200      14884  28084\n",
       "VBD              10780      11512  22292\n",
       "VB                9922      11190  21112\n",
       ".                 9743      10718  20461\n",
       "CC                8371       9271  17642\n",
       "PRP$              7218       8246  15464\n",
       "NNS               6829       7930  14759\n",
       "VBP               6881       7768  14649\n",
       ",                 5823       7155  12978\n",
       "TO                5699       6392  12091\n",
       "VBZ               5375       5827  11202\n",
       "VBG               4809       5421  10230\n",
       "VBN               3487       3841   7328\n",
       "MD                2383       2706   5089\n",
       "CD                2263       2499   4762\n",
       "RP                1459       1482   2941\n",
       "WRB               1337       1511   2848\n",
       ":                  813       1054   1867\n",
       ")                  798        934   1732\n",
       "WP                 829        875   1704\n",
       "(                  782        886   1668\n",
       "NNP                792        831   1623\n",
       "WDT                540        602   1142\n",
       "JJR                421        536    957\n",
       "''                 465        483    948\n",
       "``                 423        446    869\n",
       "JJS                327        337    664\n",
       "POS                260        309    569\n",
       "EX                 273        287    560\n",
       "RBR                247        265    512\n",
       "PDT                172        221    393\n",
       "$                  144        142    286\n",
       "FW                  82        117    199\n",
       "#                   70         96    166\n",
       "UH                  51         74    125\n",
       "RBS                 52         56    108\n",
       "SYM                  5          8     13\n",
       "WP$                  8          3     11"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import Counter\n",
    "\n",
    "counts = Counter(tag for word, tag in get_archive_texts_pos_tags(['YTA', 'ESH']))\n",
    "a_hole_pos_freq = pd.DataFrame(counts, index=[0])\n",
    "a_hole_pos_freq = a_hole_pos_freq.transpose()\n",
    "a_hole_pos_freq.columns = ['a_hole_pos_count']\n",
    "\n",
    "counts = Counter(tag for word, tag in get_archive_texts_pos_tags(['NTA', 'NAH']))\n",
    "pos_freq = pd.DataFrame(counts, index=[0])\n",
    "pos_freq = pos_freq.transpose()\n",
    "pos_freq.columns = ['pos_count']\n",
    "\n",
    "\n",
    "pos_freqs = a_hole_pos_freq.join(pos_freq)\n",
    "pos_freqs['total'] = pos_freqs['a_hole_pos_count'] + pos_freqs['pos_count']\n",
    "pos_freqs.sort_values(by=['total'], ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### N-grams and Collocations\n",
    "Lets explore making bigrams and trigrams and finding collocations for this corpus.\n",
    "\n",
    "First lets look at the collocations generated from bigrams. There are several different scoring measures that can be used as is listed in one of the comments below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bigram_word_1_a_hole</th>\n",
       "      <th>bigram_word_2_a_hole</th>\n",
       "      <th>score_a_hole</th>\n",
       "      <th>bigram_word_1</th>\n",
       "      <th>bigram_word_2</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>peanut</td>\n",
       "      <td>butter</td>\n",
       "      <td>14.027584</td>\n",
       "      <td>april</td>\n",
       "      <td>fools</td>\n",
       "      <td>14.641650</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>mr</td>\n",
       "      <td>lastname</td>\n",
       "      <td>13.995163</td>\n",
       "      <td>baked</td>\n",
       "      <td>goods</td>\n",
       "      <td>14.419257</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>passive</td>\n",
       "      <td>aggressive</td>\n",
       "      <td>13.709408</td>\n",
       "      <td>nail</td>\n",
       "      <td>polish</td>\n",
       "      <td>14.378615</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>miss</td>\n",
       "      <td>johnson</td>\n",
       "      <td>13.679661</td>\n",
       "      <td>ice</td>\n",
       "      <td>cream</td>\n",
       "      <td>13.864042</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>fake</td>\n",
       "      <td>lashes</td>\n",
       "      <td>13.487016</td>\n",
       "      <td>tl</td>\n",
       "      <td>dr</td>\n",
       "      <td>13.471725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>y</td>\n",
       "      <td>o</td>\n",
       "      <td>13.371538</td>\n",
       "      <td>imgur</td>\n",
       "      <td>com</td>\n",
       "      <td>13.471725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>tl</td>\n",
       "      <td>dr</td>\n",
       "      <td>13.317091</td>\n",
       "      <td>plant</td>\n",
       "      <td>based</td>\n",
       "      <td>13.393722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>400</td>\n",
       "      <td>calories</td>\n",
       "      <td>13.094698</td>\n",
       "      <td>cow</td>\n",
       "      <td>milk</td>\n",
       "      <td>13.356247</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>blah</td>\n",
       "      <td>blah</td>\n",
       "      <td>13.042231</td>\n",
       "      <td>passive</td>\n",
       "      <td>aggressive</td>\n",
       "      <td>13.352143</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>social</td>\n",
       "      <td>media</td>\n",
       "      <td>12.732128</td>\n",
       "      <td>https</td>\n",
       "      <td>imgur</td>\n",
       "      <td>13.319722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>ice</td>\n",
       "      <td>cream</td>\n",
       "      <td>12.700419</td>\n",
       "      <td>co</td>\n",
       "      <td>worker</td>\n",
       "      <td>13.056687</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>ha</td>\n",
       "      <td>ha</td>\n",
       "      <td>12.302591</td>\n",
       "      <td>mixed</td>\n",
       "      <td>race</td>\n",
       "      <td>12.886762</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>passport</td>\n",
       "      <td>photos</td>\n",
       "      <td>12.117782</td>\n",
       "      <td>public</td>\n",
       "      <td>transportation</td>\n",
       "      <td>12.886762</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>taken</td>\n",
       "      <td>aback</td>\n",
       "      <td>12.094698</td>\n",
       "      <td>rewards</td>\n",
       "      <td>card</td>\n",
       "      <td>12.886762</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>mental</td>\n",
       "      <td>health</td>\n",
       "      <td>12.027584</td>\n",
       "      <td>uncle</td>\n",
       "      <td>ben</td>\n",
       "      <td>12.886762</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>video</td>\n",
       "      <td>games</td>\n",
       "      <td>12.001589</td>\n",
       "      <td>computer</td>\n",
       "      <td>lab</td>\n",
       "      <td>12.876115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>gluten</td>\n",
       "      <td>free</td>\n",
       "      <td>11.902053</td>\n",
       "      <td>self</td>\n",
       "      <td>feeder</td>\n",
       "      <td>12.834295</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>c</td>\n",
       "      <td>section</td>\n",
       "      <td>11.716187</td>\n",
       "      <td>gas</td>\n",
       "      <td>station</td>\n",
       "      <td>12.793653</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>fast</td>\n",
       "      <td>forward</td>\n",
       "      <td>11.639735</td>\n",
       "      <td>social</td>\n",
       "      <td>media</td>\n",
       "      <td>12.734759</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>phd</td>\n",
       "      <td>student</td>\n",
       "      <td>11.580125</td>\n",
       "      <td>aunt</td>\n",
       "      <td>tina</td>\n",
       "      <td>12.432196</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>mouth</td>\n",
       "      <td>shut</td>\n",
       "      <td>11.532819</td>\n",
       "      <td>flu</td>\n",
       "      <td>shot</td>\n",
       "      <td>12.319722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>difference</td>\n",
       "      <td>between</td>\n",
       "      <td>10.673234</td>\n",
       "      <td>mouth</td>\n",
       "      <td>shut</td>\n",
       "      <td>12.149797</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>bachelor</td>\n",
       "      <td>party</td>\n",
       "      <td>10.650514</td>\n",
       "      <td>fast</td>\n",
       "      <td>forward</td>\n",
       "      <td>12.149797</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>birth</td>\n",
       "      <td>control</td>\n",
       "      <td>10.584942</td>\n",
       "      <td>speaking</td>\n",
       "      <td>korean</td>\n",
       "      <td>12.111829</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>watching</td>\n",
       "      <td>tv</td>\n",
       "      <td>10.478027</td>\n",
       "      <td>dress</td>\n",
       "      <td>code</td>\n",
       "      <td>11.997793</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>story</td>\n",
       "      <td>short</td>\n",
       "      <td>10.351856</td>\n",
       "      <td>customer</td>\n",
       "      <td>service</td>\n",
       "      <td>11.986298</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>24</td>\n",
       "      <td>7</td>\n",
       "      <td>10.329163</td>\n",
       "      <td>fake</td>\n",
       "      <td>accent</td>\n",
       "      <td>11.834295</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>5</td>\n",
       "      <td>30am</td>\n",
       "      <td>10.285382</td>\n",
       "      <td>video</td>\n",
       "      <td>games</td>\n",
       "      <td>11.734759</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>security</td>\n",
       "      <td>line</td>\n",
       "      <td>10.212754</td>\n",
       "      <td>staff</td>\n",
       "      <td>member</td>\n",
       "      <td>11.471725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>wedding</td>\n",
       "      <td>fund</td>\n",
       "      <td>9.857659</td>\n",
       "      <td>title</td>\n",
       "      <td>sounds</td>\n",
       "      <td>11.441977</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>dick</td>\n",
       "      <td>move</td>\n",
       "      <td>9.843159</td>\n",
       "      <td>engagement</td>\n",
       "      <td>ring</td>\n",
       "      <td>11.280193</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>elementary</td>\n",
       "      <td>school</td>\n",
       "      <td>9.835964</td>\n",
       "      <td>mental</td>\n",
       "      <td>health</td>\n",
       "      <td>11.249332</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>visibly</td>\n",
       "      <td>upset</td>\n",
       "      <td>9.818840</td>\n",
       "      <td>window</td>\n",
       "      <td>seat</td>\n",
       "      <td>11.026940</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>50</td>\n",
       "      <td>50</td>\n",
       "      <td>9.761820</td>\n",
       "      <td>throwaway</td>\n",
       "      <td>account</td>\n",
       "      <td>11.012293</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>started</td>\n",
       "      <td>noticing</td>\n",
       "      <td>9.724135</td>\n",
       "      <td>blue</td>\n",
       "      <td>eyes</td>\n",
       "      <td>10.999237</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>fair</td>\n",
       "      <td>game</td>\n",
       "      <td>9.705656</td>\n",
       "      <td>sofa</td>\n",
       "      <td>bed</td>\n",
       "      <td>10.793653</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>multiple</td>\n",
       "      <td>times</td>\n",
       "      <td>9.635267</td>\n",
       "      <td>group</td>\n",
       "      <td>chat</td>\n",
       "      <td>10.718817</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>big</td>\n",
       "      <td>deal</td>\n",
       "      <td>9.611113</td>\n",
       "      <td>aisle</td>\n",
       "      <td>seat</td>\n",
       "      <td>10.441977</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>last</td>\n",
       "      <td>straw</td>\n",
       "      <td>9.462430</td>\n",
       "      <td>50</td>\n",
       "      <td>50</td>\n",
       "      <td>10.348868</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>middle</td>\n",
       "      <td>class</td>\n",
       "      <td>9.447000</td>\n",
       "      <td>birth</td>\n",
       "      <td>control</td>\n",
       "      <td>10.310733</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>dining</td>\n",
       "      <td>room</td>\n",
       "      <td>9.439346</td>\n",
       "      <td>less</td>\n",
       "      <td>active</td>\n",
       "      <td>10.297354</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>special</td>\n",
       "      <td>needs</td>\n",
       "      <td>9.439346</td>\n",
       "      <td>college</td>\n",
       "      <td>fund</td>\n",
       "      <td>10.056687</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>passed</td>\n",
       "      <td>away</td>\n",
       "      <td>9.389736</td>\n",
       "      <td>active</td>\n",
       "      <td>side</td>\n",
       "      <td>10.050261</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>foot</td>\n",
       "      <td>down</td>\n",
       "      <td>9.364619</td>\n",
       "      <td>multiple</td>\n",
       "      <td>times</td>\n",
       "      <td>9.955149</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>year</td>\n",
       "      <td>olds</td>\n",
       "      <td>9.307107</td>\n",
       "      <td>story</td>\n",
       "      <td>short</td>\n",
       "      <td>9.857015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>looking</td>\n",
       "      <td>forward</td>\n",
       "      <td>9.301149</td>\n",
       "      <td>same</td>\n",
       "      <td>page</td>\n",
       "      <td>9.834295</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>30</td>\n",
       "      <td>minutes</td>\n",
       "      <td>9.181576</td>\n",
       "      <td>looking</td>\n",
       "      <td>forward</td>\n",
       "      <td>9.605476</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>happy</td>\n",
       "      <td>holidays</td>\n",
       "      <td>9.149840</td>\n",
       "      <td>long</td>\n",
       "      <td>term</td>\n",
       "      <td>9.535741</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>car</td>\n",
       "      <td>accident</td>\n",
       "      <td>9.134751</td>\n",
       "      <td>mutual</td>\n",
       "      <td>friends</td>\n",
       "      <td>9.484664</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>mexican</td>\n",
       "      <td>food</td>\n",
       "      <td>9.129464</td>\n",
       "      <td>rent</td>\n",
       "      <td>free</td>\n",
       "      <td>9.445252</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   bigram_word_1_a_hole bigram_word_2_a_hole  score_a_hole bigram_word_1  \\\n",
       "0                peanut               butter     14.027584         april   \n",
       "1                    mr             lastname     13.995163         baked   \n",
       "2               passive           aggressive     13.709408          nail   \n",
       "3                  miss              johnson     13.679661           ice   \n",
       "4                  fake               lashes     13.487016            tl   \n",
       "5                     y                    o     13.371538         imgur   \n",
       "6                    tl                   dr     13.317091         plant   \n",
       "7                   400             calories     13.094698           cow   \n",
       "8                  blah                 blah     13.042231       passive   \n",
       "9                social                media     12.732128         https   \n",
       "10                  ice                cream     12.700419            co   \n",
       "11                   ha                   ha     12.302591         mixed   \n",
       "12             passport               photos     12.117782        public   \n",
       "13                taken                aback     12.094698       rewards   \n",
       "14               mental               health     12.027584         uncle   \n",
       "15                video                games     12.001589      computer   \n",
       "16               gluten                 free     11.902053          self   \n",
       "17                    c              section     11.716187           gas   \n",
       "18                 fast              forward     11.639735        social   \n",
       "19                  phd              student     11.580125          aunt   \n",
       "20                mouth                 shut     11.532819           flu   \n",
       "21           difference              between     10.673234         mouth   \n",
       "22             bachelor                party     10.650514          fast   \n",
       "23                birth              control     10.584942      speaking   \n",
       "24             watching                   tv     10.478027         dress   \n",
       "25                story                short     10.351856      customer   \n",
       "26                   24                    7     10.329163          fake   \n",
       "27                    5                 30am     10.285382         video   \n",
       "28             security                 line     10.212754         staff   \n",
       "29              wedding                 fund      9.857659         title   \n",
       "30                 dick                 move      9.843159    engagement   \n",
       "31           elementary               school      9.835964        mental   \n",
       "32              visibly                upset      9.818840        window   \n",
       "33                   50                   50      9.761820     throwaway   \n",
       "34              started             noticing      9.724135          blue   \n",
       "35                 fair                 game      9.705656          sofa   \n",
       "36             multiple                times      9.635267         group   \n",
       "37                  big                 deal      9.611113         aisle   \n",
       "38                 last                straw      9.462430            50   \n",
       "39               middle                class      9.447000         birth   \n",
       "40               dining                 room      9.439346          less   \n",
       "41              special                needs      9.439346       college   \n",
       "42               passed                 away      9.389736        active   \n",
       "43                 foot                 down      9.364619      multiple   \n",
       "44                 year                 olds      9.307107         story   \n",
       "45              looking              forward      9.301149          same   \n",
       "46                   30              minutes      9.181576       looking   \n",
       "47                happy             holidays      9.149840          long   \n",
       "48                  car             accident      9.134751        mutual   \n",
       "49              mexican                 food      9.129464          rent   \n",
       "\n",
       "     bigram_word_2      score  \n",
       "0            fools  14.641650  \n",
       "1            goods  14.419257  \n",
       "2           polish  14.378615  \n",
       "3            cream  13.864042  \n",
       "4               dr  13.471725  \n",
       "5              com  13.471725  \n",
       "6            based  13.393722  \n",
       "7             milk  13.356247  \n",
       "8       aggressive  13.352143  \n",
       "9            imgur  13.319722  \n",
       "10          worker  13.056687  \n",
       "11            race  12.886762  \n",
       "12  transportation  12.886762  \n",
       "13            card  12.886762  \n",
       "14             ben  12.886762  \n",
       "15             lab  12.876115  \n",
       "16          feeder  12.834295  \n",
       "17         station  12.793653  \n",
       "18           media  12.734759  \n",
       "19            tina  12.432196  \n",
       "20            shot  12.319722  \n",
       "21            shut  12.149797  \n",
       "22         forward  12.149797  \n",
       "23          korean  12.111829  \n",
       "24            code  11.997793  \n",
       "25         service  11.986298  \n",
       "26          accent  11.834295  \n",
       "27           games  11.734759  \n",
       "28          member  11.471725  \n",
       "29          sounds  11.441977  \n",
       "30            ring  11.280193  \n",
       "31          health  11.249332  \n",
       "32            seat  11.026940  \n",
       "33         account  11.012293  \n",
       "34            eyes  10.999237  \n",
       "35             bed  10.793653  \n",
       "36            chat  10.718817  \n",
       "37            seat  10.441977  \n",
       "38              50  10.348868  \n",
       "39         control  10.310733  \n",
       "40          active  10.297354  \n",
       "41            fund  10.056687  \n",
       "42            side  10.050261  \n",
       "43           times   9.955149  \n",
       "44           short   9.857015  \n",
       "45            page   9.834295  \n",
       "46         forward   9.605476  \n",
       "47            term   9.535741  \n",
       "48         friends   9.484664  \n",
       "49            free   9.445252  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "from nltk.collocations import *\n",
    "\n",
    "# Feel free to change these values.\n",
    "n_collocs = 50\n",
    "n_freq_filter = 5\n",
    "\n",
    "bigram_measures = nltk.collocations.BigramAssocMeasures()\n",
    "# Options: pmi, likelihood_ratio, chi_sq, dice, phi_sq, etc.\n",
    "scoring_measure = bigram_measures.pmi\n",
    "\n",
    "flatten = lambda l: (l[0][0], l[0][1], l[1])\n",
    "\n",
    "finder = BigramCollocationFinder.from_words(get_archive_texts(['YTA', 'ESH'], no_punctuation=True))\n",
    "finder.apply_freq_filter(n_freq_filter)\n",
    "a_hole_colloc = pd.DataFrame(list(map(flatten, finder.score_ngrams(scoring_measure))),\n",
    "                             columns=['bigram_word_1', 'bigram_word_2', 'score'])\n",
    "\n",
    "finder = BigramCollocationFinder.from_words(get_archive_texts(['NTA', 'NAH'], no_punctuation=True))\n",
    "finder.apply_freq_filter(n_freq_filter)\n",
    "colloc = pd.DataFrame(list(map(flatten, finder.score_ngrams(scoring_measure))),\n",
    "                      columns=['bigram_word_1', 'bigram_word_2', 'score'])\n",
    "\n",
    "a_hole_colloc.join(colloc, lsuffix=\"_a_hole\").head(n_collocs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What about trigrams?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>trigram_word_1_a_hole</th>\n",
       "      <th>trigram_word_2_a_hole</th>\n",
       "      <th>trigram_word_3_a_hole</th>\n",
       "      <th>score_a_hole</th>\n",
       "      <th>trigram_word_1</th>\n",
       "      <th>trigram_word_2</th>\n",
       "      <th>trigram_word_3</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>long</td>\n",
       "      <td>story</td>\n",
       "      <td>short</td>\n",
       "      <td>21.166446</td>\n",
       "      <td>https</td>\n",
       "      <td>imgur</td>\n",
       "      <td>com</td>\n",
       "      <td>26.791446</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>on</td>\n",
       "      <td>social</td>\n",
       "      <td>media</td>\n",
       "      <td>19.234156</td>\n",
       "      <td>less</td>\n",
       "      <td>active</td>\n",
       "      <td>side</td>\n",
       "      <td>21.184116</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>at</td>\n",
       "      <td>5</td>\n",
       "      <td>30am</td>\n",
       "      <td>17.816748</td>\n",
       "      <td>long</td>\n",
       "      <td>story</td>\n",
       "      <td>short</td>\n",
       "      <td>20.640683</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>year</td>\n",
       "      <td>old</td>\n",
       "      <td>boy</td>\n",
       "      <td>17.366181</td>\n",
       "      <td>blake</td>\n",
       "      <td>s</td>\n",
       "      <td>ring</td>\n",
       "      <td>18.265686</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>my</td>\n",
       "      <td>mouth</td>\n",
       "      <td>shut</td>\n",
       "      <td>17.242991</td>\n",
       "      <td>aunt</td>\n",
       "      <td>tina</td>\n",
       "      <td>s</td>\n",
       "      <td>17.780259</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>a</td>\n",
       "      <td>phd</td>\n",
       "      <td>student</td>\n",
       "      <td>17.074910</td>\n",
       "      <td>my</td>\n",
       "      <td>mouth</td>\n",
       "      <td>shut</td>\n",
       "      <td>17.750472</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>of</td>\n",
       "      <td>ice</td>\n",
       "      <td>cream</td>\n",
       "      <td>16.977982</td>\n",
       "      <td>the</td>\n",
       "      <td>self</td>\n",
       "      <td>feeder</td>\n",
       "      <td>17.721808</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>few</td>\n",
       "      <td>days</td>\n",
       "      <td>ago</td>\n",
       "      <td>16.774776</td>\n",
       "      <td>11</td>\n",
       "      <td>year</td>\n",
       "      <td>old</td>\n",
       "      <td>16.760411</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>along</td>\n",
       "      <td>the</td>\n",
       "      <td>lines</td>\n",
       "      <td>16.594526</td>\n",
       "      <td>reddit</td>\n",
       "      <td>aita</td>\n",
       "      <td>edit</td>\n",
       "      <td>16.639795</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>6</td>\n",
       "      <td>months</td>\n",
       "      <td>ago</td>\n",
       "      <td>16.448790</td>\n",
       "      <td>under</td>\n",
       "      <td>the</td>\n",
       "      <td>impression</td>\n",
       "      <td>16.622273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>rant</td>\n",
       "      <td>about</td>\n",
       "      <td>how</td>\n",
       "      <td>16.414072</td>\n",
       "      <td>haven</td>\n",
       "      <td>t</td>\n",
       "      <td>spoken</td>\n",
       "      <td>16.452309</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>fast</td>\n",
       "      <td>forward</td>\n",
       "      <td>to</td>\n",
       "      <td>16.305375</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>minutes</td>\n",
       "      <td>16.415775</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>point</td>\n",
       "      <td>of</td>\n",
       "      <td>view</td>\n",
       "      <td>16.228331</td>\n",
       "      <td>along</td>\n",
       "      <td>the</td>\n",
       "      <td>lines</td>\n",
       "      <td>16.307231</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>pisses</td>\n",
       "      <td>me</td>\n",
       "      <td>off</td>\n",
       "      <td>15.967957</td>\n",
       "      <td>the</td>\n",
       "      <td>window</td>\n",
       "      <td>seat</td>\n",
       "      <td>16.177488</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>behind</td>\n",
       "      <td>the</td>\n",
       "      <td>counter</td>\n",
       "      <td>15.966495</td>\n",
       "      <td>few</td>\n",
       "      <td>days</td>\n",
       "      <td>ago</td>\n",
       "      <td>16.019255</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>see</td>\n",
       "      <td>each</td>\n",
       "      <td>other</td>\n",
       "      <td>15.803578</td>\n",
       "      <td>couple</td>\n",
       "      <td>months</td>\n",
       "      <td>ago</td>\n",
       "      <td>15.964430</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>10</td>\n",
       "      <td>year</td>\n",
       "      <td>old</td>\n",
       "      <td>15.781218</td>\n",
       "      <td>fast</td>\n",
       "      <td>forward</td>\n",
       "      <td>to</td>\n",
       "      <td>15.929487</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>lost</td>\n",
       "      <td>his</td>\n",
       "      <td>job</td>\n",
       "      <td>15.677230</td>\n",
       "      <td>5</td>\n",
       "      <td>year</td>\n",
       "      <td>old</td>\n",
       "      <td>15.779520</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>mother</td>\n",
       "      <td>in</td>\n",
       "      <td>law</td>\n",
       "      <td>15.639123</td>\n",
       "      <td>couldn</td>\n",
       "      <td>t</td>\n",
       "      <td>afford</td>\n",
       "      <td>15.607939</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>5</td>\n",
       "      <td>year</td>\n",
       "      <td>old</td>\n",
       "      <td>15.497971</td>\n",
       "      <td>part</td>\n",
       "      <td>time</td>\n",
       "      <td>job</td>\n",
       "      <td>15.574314</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>did</td>\n",
       "      <td>anything</td>\n",
       "      <td>wrong</td>\n",
       "      <td>15.457962</td>\n",
       "      <td>two</td>\n",
       "      <td>days</td>\n",
       "      <td>ago</td>\n",
       "      <td>15.564689</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>edit</td>\n",
       "      <td>thank</td>\n",
       "      <td>you</td>\n",
       "      <td>15.389394</td>\n",
       "      <td>full</td>\n",
       "      <td>time</td>\n",
       "      <td>job</td>\n",
       "      <td>15.435416</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>the</td>\n",
       "      <td>security</td>\n",
       "      <td>line</td>\n",
       "      <td>15.364658</td>\n",
       "      <td>was</td>\n",
       "      <td>looking</td>\n",
       "      <td>forward</td>\n",
       "      <td>15.430142</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>6</td>\n",
       "      <td>year</td>\n",
       "      <td>old</td>\n",
       "      <td>15.282765</td>\n",
       "      <td>m</td>\n",
       "      <td>being</td>\n",
       "      <td>unreasonable</td>\n",
       "      <td>15.352987</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>ice</td>\n",
       "      <td>cream</td>\n",
       "      <td>i</td>\n",
       "      <td>15.169757</td>\n",
       "      <td>few</td>\n",
       "      <td>months</td>\n",
       "      <td>ago</td>\n",
       "      <td>15.306404</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>an</td>\n",
       "      <td>hour</td>\n",
       "      <td>ago</td>\n",
       "      <td>15.069216</td>\n",
       "      <td>two</td>\n",
       "      <td>months</td>\n",
       "      <td>ago</td>\n",
       "      <td>15.214408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>year</td>\n",
       "      <td>old</td>\n",
       "      <td>son</td>\n",
       "      <td>14.995998</td>\n",
       "      <td>for</td>\n",
       "      <td>your</td>\n",
       "      <td>input</td>\n",
       "      <td>15.195482</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>part</td>\n",
       "      <td>time</td>\n",
       "      <td>job</td>\n",
       "      <td>14.992999</td>\n",
       "      <td>hour</td>\n",
       "      <td>or</td>\n",
       "      <td>two</td>\n",
       "      <td>15.186816</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>year</td>\n",
       "      <td>old</td>\n",
       "      <td>daughter</td>\n",
       "      <td>14.936000</td>\n",
       "      <td>the</td>\n",
       "      <td>less</td>\n",
       "      <td>active</td>\n",
       "      <td>15.184867</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>two</td>\n",
       "      <td>years</td>\n",
       "      <td>ago</td>\n",
       "      <td>14.904976</td>\n",
       "      <td>2</td>\n",
       "      <td>months</td>\n",
       "      <td>ago</td>\n",
       "      <td>15.169620</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>put</td>\n",
       "      <td>my</td>\n",
       "      <td>foot</td>\n",
       "      <td>14.850230</td>\n",
       "      <td>year</td>\n",
       "      <td>old</td>\n",
       "      <td>daughter</td>\n",
       "      <td>15.139516</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>my</td>\n",
       "      <td>foot</td>\n",
       "      <td>down</td>\n",
       "      <td>14.811756</td>\n",
       "      <td>thank</td>\n",
       "      <td>you</td>\n",
       "      <td>everyone</td>\n",
       "      <td>15.138347</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>had</td>\n",
       "      <td>no</td>\n",
       "      <td>idea</td>\n",
       "      <td>14.811362</td>\n",
       "      <td>the</td>\n",
       "      <td>aisle</td>\n",
       "      <td>seat</td>\n",
       "      <td>15.107099</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>a</td>\n",
       "      <td>big</td>\n",
       "      <td>deal</td>\n",
       "      <td>14.783970</td>\n",
       "      <td>year</td>\n",
       "      <td>old</td>\n",
       "      <td>son</td>\n",
       "      <td>15.092886</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>as</td>\n",
       "      <td>soon</td>\n",
       "      <td>as</td>\n",
       "      <td>14.760553</td>\n",
       "      <td>figure</td>\n",
       "      <td>out</td>\n",
       "      <td>what</td>\n",
       "      <td>15.011596</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>few</td>\n",
       "      <td>years</td>\n",
       "      <td>ago</td>\n",
       "      <td>14.685967</td>\n",
       "      <td>getting</td>\n",
       "      <td>rid</td>\n",
       "      <td>of</td>\n",
       "      <td>14.998054</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>needless</td>\n",
       "      <td>to</td>\n",
       "      <td>say</td>\n",
       "      <td>14.630215</td>\n",
       "      <td>so</td>\n",
       "      <td>reddit</td>\n",
       "      <td>aita</td>\n",
       "      <td>14.987602</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>sister</td>\n",
       "      <td>in</td>\n",
       "      <td>law</td>\n",
       "      <td>14.627918</td>\n",
       "      <td>the</td>\n",
       "      <td>same</td>\n",
       "      <td>page</td>\n",
       "      <td>14.984843</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>brother</td>\n",
       "      <td>in</td>\n",
       "      <td>law</td>\n",
       "      <td>14.499512</td>\n",
       "      <td>no</td>\n",
       "      <td>matter</td>\n",
       "      <td>what</td>\n",
       "      <td>14.972387</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>2</td>\n",
       "      <td>years</td>\n",
       "      <td>ago</td>\n",
       "      <td>14.427455</td>\n",
       "      <td>few</td>\n",
       "      <td>years</td>\n",
       "      <td>ago</td>\n",
       "      <td>14.971497</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>so</td>\n",
       "      <td>reddit</td>\n",
       "      <td>aita</td>\n",
       "      <td>14.410614</td>\n",
       "      <td>haven</td>\n",
       "      <td>t</td>\n",
       "      <td>seen</td>\n",
       "      <td>14.867346</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>making</td>\n",
       "      <td>fun</td>\n",
       "      <td>of</td>\n",
       "      <td>14.320450</td>\n",
       "      <td>every</td>\n",
       "      <td>other</td>\n",
       "      <td>week</td>\n",
       "      <td>14.827157</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>looking</td>\n",
       "      <td>forward</td>\n",
       "      <td>to</td>\n",
       "      <td>14.314713</td>\n",
       "      <td>wasn</td>\n",
       "      <td>t</td>\n",
       "      <td>expecting</td>\n",
       "      <td>14.696587</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>a</td>\n",
       "      <td>college</td>\n",
       "      <td>student</td>\n",
       "      <td>14.247091</td>\n",
       "      <td>looking</td>\n",
       "      <td>forward</td>\n",
       "      <td>to</td>\n",
       "      <td>14.607559</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>back</td>\n",
       "      <td>and</td>\n",
       "      <td>forth</td>\n",
       "      <td>14.213867</td>\n",
       "      <td>this</td>\n",
       "      <td>whole</td>\n",
       "      <td>situation</td>\n",
       "      <td>14.590430</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>would</td>\n",
       "      <td>be</td>\n",
       "      <td>funny</td>\n",
       "      <td>14.190002</td>\n",
       "      <td>ended</td>\n",
       "      <td>up</td>\n",
       "      <td>getting</td>\n",
       "      <td>14.508265</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>work</td>\n",
       "      <td>full</td>\n",
       "      <td>time</td>\n",
       "      <td>14.135433</td>\n",
       "      <td>two</td>\n",
       "      <td>years</td>\n",
       "      <td>ago</td>\n",
       "      <td>14.490459</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>use</td>\n",
       "      <td>the</td>\n",
       "      <td>bathroom</td>\n",
       "      <td>13.979816</td>\n",
       "      <td>can</td>\n",
       "      <td>t</td>\n",
       "      <td>imagine</td>\n",
       "      <td>14.435500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>couldn</td>\n",
       "      <td>t</td>\n",
       "      <td>help</td>\n",
       "      <td>13.975685</td>\n",
       "      <td>sister</td>\n",
       "      <td>in</td>\n",
       "      <td>law</td>\n",
       "      <td>14.341859</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>in</td>\n",
       "      <td>high</td>\n",
       "      <td>school</td>\n",
       "      <td>13.866765</td>\n",
       "      <td>as</td>\n",
       "      <td>soon</td>\n",
       "      <td>as</td>\n",
       "      <td>14.328839</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   trigram_word_1_a_hole trigram_word_2_a_hole trigram_word_3_a_hole  \\\n",
       "0                   long                 story                 short   \n",
       "1                     on                social                 media   \n",
       "2                     at                     5                  30am   \n",
       "3                   year                   old                   boy   \n",
       "4                     my                 mouth                  shut   \n",
       "5                      a                   phd               student   \n",
       "6                     of                   ice                 cream   \n",
       "7                    few                  days                   ago   \n",
       "8                  along                   the                 lines   \n",
       "9                      6                months                   ago   \n",
       "10                  rant                 about                   how   \n",
       "11                  fast               forward                    to   \n",
       "12                 point                    of                  view   \n",
       "13                pisses                    me                   off   \n",
       "14                behind                   the               counter   \n",
       "15                   see                  each                 other   \n",
       "16                    10                  year                   old   \n",
       "17                  lost                   his                   job   \n",
       "18                mother                    in                   law   \n",
       "19                     5                  year                   old   \n",
       "20                   did              anything                 wrong   \n",
       "21                  edit                 thank                   you   \n",
       "22                   the              security                  line   \n",
       "23                     6                  year                   old   \n",
       "24                   ice                 cream                     i   \n",
       "25                    an                  hour                   ago   \n",
       "26                  year                   old                   son   \n",
       "27                  part                  time                   job   \n",
       "28                  year                   old              daughter   \n",
       "29                   two                 years                   ago   \n",
       "30                   put                    my                  foot   \n",
       "31                    my                  foot                  down   \n",
       "32                   had                    no                  idea   \n",
       "33                     a                   big                  deal   \n",
       "34                    as                  soon                    as   \n",
       "35                   few                 years                   ago   \n",
       "36              needless                    to                   say   \n",
       "37                sister                    in                   law   \n",
       "38               brother                    in                   law   \n",
       "39                     2                 years                   ago   \n",
       "40                    so                reddit                  aita   \n",
       "41                making                   fun                    of   \n",
       "42               looking               forward                    to   \n",
       "43                     a               college               student   \n",
       "44                  back                   and                 forth   \n",
       "45                 would                    be                 funny   \n",
       "46                  work                  full                  time   \n",
       "47                   use                   the              bathroom   \n",
       "48                couldn                     t                  help   \n",
       "49                    in                  high                school   \n",
       "\n",
       "    score_a_hole trigram_word_1 trigram_word_2 trigram_word_3      score  \n",
       "0      21.166446          https          imgur            com  26.791446  \n",
       "1      19.234156           less         active           side  21.184116  \n",
       "2      17.816748           long          story          short  20.640683  \n",
       "3      17.366181          blake              s           ring  18.265686  \n",
       "4      17.242991           aunt           tina              s  17.780259  \n",
       "5      17.074910             my          mouth           shut  17.750472  \n",
       "6      16.977982            the           self         feeder  17.721808  \n",
       "7      16.774776             11           year            old  16.760411  \n",
       "8      16.594526         reddit           aita           edit  16.639795  \n",
       "9      16.448790          under            the     impression  16.622273  \n",
       "10     16.414072          haven              t         spoken  16.452309  \n",
       "11     16.305375              2              3        minutes  16.415775  \n",
       "12     16.228331          along            the          lines  16.307231  \n",
       "13     15.967957            the         window           seat  16.177488  \n",
       "14     15.966495            few           days            ago  16.019255  \n",
       "15     15.803578         couple         months            ago  15.964430  \n",
       "16     15.781218           fast        forward             to  15.929487  \n",
       "17     15.677230              5           year            old  15.779520  \n",
       "18     15.639123         couldn              t         afford  15.607939  \n",
       "19     15.497971           part           time            job  15.574314  \n",
       "20     15.457962            two           days            ago  15.564689  \n",
       "21     15.389394           full           time            job  15.435416  \n",
       "22     15.364658            was        looking        forward  15.430142  \n",
       "23     15.282765              m          being   unreasonable  15.352987  \n",
       "24     15.169757            few         months            ago  15.306404  \n",
       "25     15.069216            two         months            ago  15.214408  \n",
       "26     14.995998            for           your          input  15.195482  \n",
       "27     14.992999           hour             or            two  15.186816  \n",
       "28     14.936000            the           less         active  15.184867  \n",
       "29     14.904976              2         months            ago  15.169620  \n",
       "30     14.850230           year            old       daughter  15.139516  \n",
       "31     14.811756          thank            you       everyone  15.138347  \n",
       "32     14.811362            the          aisle           seat  15.107099  \n",
       "33     14.783970           year            old            son  15.092886  \n",
       "34     14.760553         figure            out           what  15.011596  \n",
       "35     14.685967        getting            rid             of  14.998054  \n",
       "36     14.630215             so         reddit           aita  14.987602  \n",
       "37     14.627918            the           same           page  14.984843  \n",
       "38     14.499512             no         matter           what  14.972387  \n",
       "39     14.427455            few          years            ago  14.971497  \n",
       "40     14.410614          haven              t           seen  14.867346  \n",
       "41     14.320450          every          other           week  14.827157  \n",
       "42     14.314713           wasn              t      expecting  14.696587  \n",
       "43     14.247091        looking        forward             to  14.607559  \n",
       "44     14.213867           this          whole      situation  14.590430  \n",
       "45     14.190002          ended             up        getting  14.508265  \n",
       "46     14.135433            two          years            ago  14.490459  \n",
       "47     13.979816            can              t        imagine  14.435500  \n",
       "48     13.975685         sister             in            law  14.341859  \n",
       "49     13.866765             as           soon             as  14.328839  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "from nltk.collocations import *\n",
    "\n",
    "# Feel free to change these values.\n",
    "n_collocs = 50\n",
    "n_freq_filter = 5\n",
    "\n",
    "trigram_measures = nltk.collocations.TrigramAssocMeasures()\n",
    "# Options: pmi, raw_freq, likelihood_ratio, chi_sq, jaccard, dice, phi_sq, etc.\n",
    "scoring_measure = trigram_measures.pmi\n",
    "\n",
    "flatten = lambda l: (l[0][0], l[0][1], l[0][2], l[1])\n",
    "\n",
    "finder = TrigramCollocationFinder.from_words(get_archive_texts(['YTA', 'ESH'], no_punctuation=True))\n",
    "finder.apply_freq_filter(n_freq_filter)\n",
    "a_hole_colloc = pd.DataFrame(list(map(flatten, finder.score_ngrams(scoring_measure)[:n_collocs])),\n",
    "                             columns=['trigram_word_1', 'trigram_word_2', 'trigram_word_3', 'score'])\n",
    "\n",
    "finder = TrigramCollocationFinder.from_words(get_archive_texts(['NTA', 'NAH'], no_punctuation=True))\n",
    "finder.apply_freq_filter(n_freq_filter)\n",
    "colloc = pd.DataFrame(list(map(flatten, finder.score_ngrams(scoring_measure)[:n_collocs])),\n",
    "                      columns=['trigram_word_1', 'trigram_word_2', 'trigram_word_3', 'score'])\n",
    "\n",
    "a_hole_colloc.join(colloc, lsuffix=\"_a_hole\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sentiment Analysis\n",
    "NLTK provides a sentiment analysis model already called [VADER](https://github.com/cjhutto/vaderSentiment):\n",
    "\n",
    "\"VADER (Valence Aware Dictionary and sEntiment Reasoner) is a lexicon and rule-based sentiment analysis tool that is specifically attuned to sentiments expressed in social media.\"\n",
    "\n",
    "The following are some functions to help process the text with the sentiment analyzer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.sentiment.vader import SentimentIntensityAnalyzer\n",
    "\n",
    "def get_average_sentiment_scores(text):\n",
    "    '''\n",
    "    Takes in a list of sentences to get the averge sentiment\n",
    "    scores for each sentence using VADER.\n",
    "    '''\n",
    "    sid = SentimentIntensityAnalyzer()\n",
    "    averages = {'neg': 0.0, 'neu': 0.0, 'pos': 0.0, 'compound': 0.0}\n",
    "    count = 0\n",
    "    for sentence in text:\n",
    "        ss = sid.polarity_scores(sentence)\n",
    "        for key, val in ss.items():\n",
    "            averages[key] += val\n",
    "        count += 1\n",
    "    for key in averages.keys():\n",
    "        averages[key] /= count\n",
    "    return averages\n",
    "\n",
    "def corpus_sentiment_scores():\n",
    "    '''\n",
    "    Updates the corpus with the average sentiment scores of each post.\n",
    "    '''\n",
    "    global corpus\n",
    "    sid = SentimentIntensityAnalyzer()\n",
    "    scores = list(map(get_average_sentiment_scores, corpus['sent_tokens']))\n",
    "    corpus = corpus[corpus.columns.difference(scores[0].keys())].join(\n",
    "        pd.DataFrame(list(map(lambda d : d.values(), scores)), columns=scores[0].keys()))\n",
    "    \n",
    "def average_archive_sentiment_scores(archives):\n",
    "    '''\n",
    "    Gets the average sentiment scores for each archive and returns it\n",
    "    as a dataframe.\n",
    "    '''\n",
    "    global corpus\n",
    "    sentiment_scores = corpus.loc[\n",
    "        list(map(lambda row: row[1]['archive'] in archives,\n",
    "                 list(corpus.iterrows())))]\n",
    "    return sentiment_scores.mean(axis=0).to_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets first put these scores into the corpus dataframe. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>archive</th>\n",
       "      <th>id</th>\n",
       "      <th>pos_tags</th>\n",
       "      <th>selftext</th>\n",
       "      <th>sent_tokens</th>\n",
       "      <th>title</th>\n",
       "      <th>tokens</th>\n",
       "      <th>tokens_no_punct</th>\n",
       "      <th>url</th>\n",
       "      <th>neg</th>\n",
       "      <th>neu</th>\n",
       "      <th>pos</th>\n",
       "      <th>compound</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>YTA</td>\n",
       "      <td>e8l9h6</td>\n",
       "      <td>[(my, PRP$), (husband, NN), ((, (), (30m, CD),...</td>\n",
       "      <td>My husband (30M) is defending his PhD in 1 wee...</td>\n",
       "      <td>[My husband (30M) is defending his PhD in 1 we...</td>\n",
       "      <td>WIBTA if I didn't go to my husband's PhD defense</td>\n",
       "      <td>[my, husband, (, 30m, ), is, defending, his, p...</td>\n",
       "      <td>[my, husband, 30m, is, defending, his, phd, in...</td>\n",
       "      <td>https://www.reddit.com/r/AmItheAsshole/comment...</td>\n",
       "      <td>0.074684</td>\n",
       "      <td>0.819947</td>\n",
       "      <td>0.105368</td>\n",
       "      <td>0.099732</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>YTA</td>\n",
       "      <td>ec6a0d</td>\n",
       "      <td>[(okay, NN), (,, ,), (to, TO), (establish, VB)...</td>\n",
       "      <td>Okay, to establish this, my ex and I have been...</td>\n",
       "      <td>[Okay, to establish this, my ex and I have bee...</td>\n",
       "      <td>AITA for not offering to help my ex pay for di...</td>\n",
       "      <td>[okay, ,, to, establish, this, ,, my, ex, and,...</td>\n",
       "      <td>[okay, to, establish, this, my, ex, and, i, ha...</td>\n",
       "      <td>https://www.reddit.com/r/AmItheAsshole/comment...</td>\n",
       "      <td>0.046938</td>\n",
       "      <td>0.851500</td>\n",
       "      <td>0.101625</td>\n",
       "      <td>0.061269</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>YTA</td>\n",
       "      <td>duxns4</td>\n",
       "      <td>[(so, RB), (thursday, JJ), (,, ,), (i, JJ), (w...</td>\n",
       "      <td>So Thursday, I went to a party one of my frien...</td>\n",
       "      <td>[So Thursday, I went to a party one of my frie...</td>\n",
       "      <td>AITA for being livid at my roommate for callin...</td>\n",
       "      <td>[so, thursday, ,, i, went, to, a, party, one, ...</td>\n",
       "      <td>[so, thursday, i, went, to, a, party, one, of,...</td>\n",
       "      <td>https://www.reddit.com/r/AmItheAsshole/comment...</td>\n",
       "      <td>0.073577</td>\n",
       "      <td>0.864731</td>\n",
       "      <td>0.061615</td>\n",
       "      <td>-0.061638</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>YTA</td>\n",
       "      <td>e617k1</td>\n",
       "      <td>[(i, RB), (wanted, VBD), (to, TO), (do, VB), (...</td>\n",
       "      <td>I wanted to do something nice for my mom for h...</td>\n",
       "      <td>[I wanted to do something nice for my mom for ...</td>\n",
       "      <td>AITA for not painting my step father in a fami...</td>\n",
       "      <td>[i, wanted, to, do, something, nice, for, my, ...</td>\n",
       "      <td>[i, wanted, to, do, something, nice, for, my, ...</td>\n",
       "      <td>https://www.reddit.com/r/AmItheAsshole/comment...</td>\n",
       "      <td>0.047857</td>\n",
       "      <td>0.851571</td>\n",
       "      <td>0.100571</td>\n",
       "      <td>0.095214</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>YTA</td>\n",
       "      <td>e8tpjf</td>\n",
       "      <td>[(edit, NN), (:, :), (threw, NN), (out*, VBZ),...</td>\n",
       "      <td>Edit: threw OUT* a diaper beside my head\\n\\nTh...</td>\n",
       "      <td>[Edit: threw OUT* a diaper beside my head\\n\\nT...</td>\n",
       "      <td>AITA for being mad someone threw a diaper next...</td>\n",
       "      <td>[edit, :, threw, out*, a, diaper, beside, my, ...</td>\n",
       "      <td>[edit, threw, out, a, diaper, beside, my, head...</td>\n",
       "      <td>https://www.reddit.com/r/AmItheAsshole/comment...</td>\n",
       "      <td>0.072667</td>\n",
       "      <td>0.834185</td>\n",
       "      <td>0.093185</td>\n",
       "      <td>0.014785</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  archive      id                                           pos_tags  \\\n",
       "0     YTA  e8l9h6  [(my, PRP$), (husband, NN), ((, (), (30m, CD),...   \n",
       "1     YTA  ec6a0d  [(okay, NN), (,, ,), (to, TO), (establish, VB)...   \n",
       "2     YTA  duxns4  [(so, RB), (thursday, JJ), (,, ,), (i, JJ), (w...   \n",
       "3     YTA  e617k1  [(i, RB), (wanted, VBD), (to, TO), (do, VB), (...   \n",
       "4     YTA  e8tpjf  [(edit, NN), (:, :), (threw, NN), (out*, VBZ),...   \n",
       "\n",
       "                                            selftext  \\\n",
       "0  My husband (30M) is defending his PhD in 1 wee...   \n",
       "1  Okay, to establish this, my ex and I have been...   \n",
       "2  So Thursday, I went to a party one of my frien...   \n",
       "3  I wanted to do something nice for my mom for h...   \n",
       "4  Edit: threw OUT* a diaper beside my head\\n\\nTh...   \n",
       "\n",
       "                                         sent_tokens  \\\n",
       "0  [My husband (30M) is defending his PhD in 1 we...   \n",
       "1  [Okay, to establish this, my ex and I have bee...   \n",
       "2  [So Thursday, I went to a party one of my frie...   \n",
       "3  [I wanted to do something nice for my mom for ...   \n",
       "4  [Edit: threw OUT* a diaper beside my head\\n\\nT...   \n",
       "\n",
       "                                               title  \\\n",
       "0   WIBTA if I didn't go to my husband's PhD defense   \n",
       "1  AITA for not offering to help my ex pay for di...   \n",
       "2  AITA for being livid at my roommate for callin...   \n",
       "3  AITA for not painting my step father in a fami...   \n",
       "4  AITA for being mad someone threw a diaper next...   \n",
       "\n",
       "                                              tokens  \\\n",
       "0  [my, husband, (, 30m, ), is, defending, his, p...   \n",
       "1  [okay, ,, to, establish, this, ,, my, ex, and,...   \n",
       "2  [so, thursday, ,, i, went, to, a, party, one, ...   \n",
       "3  [i, wanted, to, do, something, nice, for, my, ...   \n",
       "4  [edit, :, threw, out*, a, diaper, beside, my, ...   \n",
       "\n",
       "                                     tokens_no_punct  \\\n",
       "0  [my, husband, 30m, is, defending, his, phd, in...   \n",
       "1  [okay, to, establish, this, my, ex, and, i, ha...   \n",
       "2  [so, thursday, i, went, to, a, party, one, of,...   \n",
       "3  [i, wanted, to, do, something, nice, for, my, ...   \n",
       "4  [edit, threw, out, a, diaper, beside, my, head...   \n",
       "\n",
       "                                                 url       neg       neu  \\\n",
       "0  https://www.reddit.com/r/AmItheAsshole/comment...  0.074684  0.819947   \n",
       "1  https://www.reddit.com/r/AmItheAsshole/comment...  0.046938  0.851500   \n",
       "2  https://www.reddit.com/r/AmItheAsshole/comment...  0.073577  0.864731   \n",
       "3  https://www.reddit.com/r/AmItheAsshole/comment...  0.047857  0.851571   \n",
       "4  https://www.reddit.com/r/AmItheAsshole/comment...  0.072667  0.834185   \n",
       "\n",
       "        pos  compound  \n",
       "0  0.105368  0.099732  \n",
       "1  0.101625  0.061269  \n",
       "2  0.061615 -0.061638  \n",
       "3  0.100571  0.095214  \n",
       "4  0.093185  0.014785  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Preprocessing\n",
    "corpus_sentiment_scores()\n",
    "corpus.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How do sentiment scores between the different judgemnts compare on average?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>neg</th>\n",
       "      <th>neu</th>\n",
       "      <th>pos</th>\n",
       "      <th>compound</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Asshole</th>\n",
       "      <td>0.073949</td>\n",
       "      <td>0.837025</td>\n",
       "      <td>0.088843</td>\n",
       "      <td>0.042957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Not the A-hole</th>\n",
       "      <td>0.079794</td>\n",
       "      <td>0.838691</td>\n",
       "      <td>0.080870</td>\n",
       "      <td>0.008572</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Everyone Sucks</th>\n",
       "      <td>0.077294</td>\n",
       "      <td>0.840701</td>\n",
       "      <td>0.081026</td>\n",
       "      <td>0.003074</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>No A-holes here</th>\n",
       "      <td>0.071918</td>\n",
       "      <td>0.833326</td>\n",
       "      <td>0.093873</td>\n",
       "      <td>0.052523</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      neg       neu       pos  compound\n",
       "Asshole          0.073949  0.837025  0.088843  0.042957\n",
       "Not the A-hole   0.079794  0.838691  0.080870  0.008572\n",
       "Everyone Sucks   0.077294  0.840701  0.081026  0.003074\n",
       "No A-holes here  0.071918  0.833326  0.093873  0.052523"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentiment_scores = pd.concat(\n",
    "    [pd.DataFrame(average_archive_sentiment_scores([archive]), index=[0]) for archive in archive_names.keys()],\n",
    "    ignore_index=True)\n",
    "\n",
    "archive_abr = list(archive_names.keys())\n",
    "sentiment_scores.rename(index=lambda i : archive_names[archive_abr[i]], inplace=True)\n",
    "sentiment_scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}