clean-ramayana.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Code to clean Ramayanam in Python 3\n",
    "\n",
    "We use this code as a way to clean the files for publishing as a clean and nice epub document.\n",
    "\n",
    "### Use beautiful soup for trial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ORIG_FILE = \"Vaalmeeki-AranyaKaanda-Sargas-1-75-Devanagari-WM.html\"\n",
    "CLEAN_FILE = \"Aranyakaanda.html\"\n",
    "\n",
    "SHLOKA_TXT = r\"color:#BD0000;font-size:11pt;line-height:1.5em;\"\n",
    "MEANING_TXT = r\"text-align:center;margin-top:8px;color:#000099;font-size:12px;line-height:1.5em; \"\n",
    "COMMENT_TXT = r\" text-align:center;margin-top:8px;color:#866;font-size:11px;line-height:1.2em;\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "\n",
    "html_text = open(ORIG_FILE, \"r\", encoding=\"utf-8\").read()\n",
    "orig_soup = BeautifulSoup(html_text, \"lxml\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Strategy 1\n",
    "\n",
    "We divide the document to rows. When there is only one row it is a chapter. Else the first column is the shloka number, the second is the shloka and meaning, and the last column is the commentary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def process_rows(row):\n",
    "    col = row.find_all('td')\n",
    "    nc = len(col)\n",
    "    if nc == 1:\n",
    "        for c in col:\n",
    "            for i, d in enumerate(c.find_all('div')):\n",
    "                if i == 0:\n",
    "                    d['class'] = \"chapter\"\n",
    "    else:\n",
    "        for i, c in enumerate(col):\n",
    "            if i == 0:\n",
    "                c['class'] = \"shloka-number\"\n",
    "            elif i == 1:\n",
    "                c['class'] = \"shloka-meaning\"\n",
    "            else:\n",
    "                c['class'] = \"commentary\"\n",
    "\n",
    "my_rows = orig_soup.find_all('tr')\n",
    "\n",
    "for r in my_rows:\n",
    "    process_rows(r)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Strategy 2\n",
    "\n",
    "Our strategy is then to first convert all the table like formatting _tbody_, _td_ and _tr_ to _div_ and then use the style elements to figure out which of the css classes we have they belong to"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "table = orig_soup.find(\"table\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'table'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "table.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "table.name = 'div'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "rows = orig_soup.find_all(\"tr\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for r in rows:\n",
    "    r.name = 'div'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cols = orig_soup.find_all(\"td\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for c in cols:\n",
    "    c.name = \"div\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tbody = orig_soup.find(\"tbody\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tbody.name = \"div\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "divs = orig_soup.find_all('div')\n",
    "\n",
    "for d in divs:\n",
    "    # Shloka\n",
    "    if d.get('style') == SHLOKA_TXT:\n",
    "        d['class'] = \"shloka\"\n",
    "    # English meaning\n",
    "    elif d.get('style') == MEANING_TXT:\n",
    "        d['class'] = \"english-meaning\"\n",
    "    # Footnote\n",
    "    elif d.get('style') == COMMENT_TXT:\n",
    "        d['class'] = \"english-footnote\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deleting all the unnecassary tags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def remove_attrs(soup, whitelist=tuple()):\n",
    "    for tag in soup.findAll(True):\n",
    "        for attr in [attr for attr in tag.attrs if attr not in whitelist]:\n",
    "            del tag[attr]\n",
    "    return soup\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "clean_soup = remove_attrs(orig_soup, whitelist=('class'))\n",
    "\n",
    "head = clean_soup.head\n",
    "new_tag = clean_soup.new_tag(\"link\", rel=\"stylesheet\", href=\"ramayanam.css\")\n",
    "head.insert(0, new_tag)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Trying to clean up the shloka part with the _br_ tag when not needed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1173305"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f = open(CLEAN_FILE, \"w\", encoding=\"utf-8\")\n",
    "f.write(str(clean_soup))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}