Text Summarization of NLP

Text Summarization/summarization-with-wine-reviews-using-spacy.ipynb

@@ -0,0 +1 @@
+{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"# &#127916; Introdruction Wine Reviews\n![Imgur](https://i.imgur.com/0GFdU23.png)\n> In this notebook, I will try to explore the Wine Reviews Dataset. It contains 130k  of reviews  in Wine Reviews. And at the end of this notebook, I will try to make simple text summarizer that will summarize given reviews. The summarized reviews can be used as a reviews title also.I will use Spacy as natural language processing library for handling this project."},{"metadata":{"_uuid":"8bd7a67c45605283a384c7ed3f1108d63c6341ce"},"cell_type":"markdown","source":"## &#128203; Object Of This Project \nThe objective of this project is to build a model that can create relevant summaries for reviews written on Wine reviews. This dataset contains above 130k  reviews, and is hosted on [Kaggle](https://www.kaggle.com/zynicide/wine-reviews)."},{"metadata":{"_uuid":"55579f5b396b1d6151fe0743a4156da1f4e2429b"},"cell_type":"markdown","source":"## What Is Text Summarization?\n![Imgur](https://i.imgur.com/LLfNlBS.png)\n> Text summarization is the process of distilling the most important information from a source (or sources) to produce an abridged version for a particular user (or users) and task (or tasks)."},{"metadata":{"_uuid":"a603f52d84bce68c606a8ba684f08908e935d812"},"cell_type":"markdown","source":"## Types of Text Summarization Methods\nText summarization methods can be classified into different types.\n![Imgur](https://i.imgur.com/J5KyMBJ.png)\n**i. Based on input type:**\n\n1. Single Document, where the input length is short. Many of the early summarization systems dealt with single document summarization.\n\n2. Multi Document, where the input can be arbitrarily long.\n\n**ii. Based on the purpose:**\n\n1. Generic, where the model makes no assumptions about the domain or content of the text to be summarized and treats all inputs as homogeneous. The majority of the work that has been done revolves around generic summarization.\n\n2. Domain-specific, where the model uses domain-specific knowledge to form a more accurate summary. For example, summarizing research papers of a specific domain, biomedical documents, etc.\n\n3. Query-based, where the summary only contains information which answers natural language questions about the input text.\n\n**iii. Based on output type:**\n\n1. Extractive, where important sentences are selected from the input text to form a summary. Most summarization approaches today are extractive in nature.\n\n2. Abstractive, where the model forms its own phrases and sentences to offer a more coherent summary, like what a human would generate. This approach is definitely a more appealing, but much more difficult than extractive summarization."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"# 1. Import Packages \n"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"2860a7dca6861d4a750915c770365e06a1408ce2"},"cell_type":"code","source":"import numpy as np # linear algebra\nimport spacy\nnlp = spacy.load('en_core_web_sm')\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom wordcloud import WordCloud\nfrom IPython.display import display\nimport base64\nimport string\nimport re\nfrom collections import Counter\nfrom time import time\n# from sklearn.feature_extraction.stop_words import ENGLISH_STOP_WORDS as stopwords\nfrom nltk.corpus import stopwords\nimport nltk\nimport heapq\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.tools as tls\n%matplotlib inline\n\nstopwords = stopwords.words('english')\nsns.set_context('notebook')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5e261260676c97cde87e171b9b049763740f63aa"},"cell_type":"markdown","source":"# 2. Import Dataset \n> In this section, I will load the desired dataset for this notebook. This dataset has huge number of reviews. It will be hard to work with full dataset. So I will randomly sample the dataset into smaller chunks for easy purpose."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"a53b6190504ccf397dd408b5882cfed485046b3e"},"cell_type":"code","source":"reviews = pd.read_csv(\"../input/winemag-data-130k-v2.csv\", nrows=5000,usecols =['points', 'title', 'description'],encoding='latin1')\nreviews = reviews.dropna()\nreviews.head(15)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"489093d68a8c094167a77cdb7479c184b3e20e87"},"cell_type":"markdown","source":"# 3. Text preprocessing\n> In this step, I will be using Spacy for preprocessing text, in others words I will clearing not useful features from reviews title like punctuation, stopwords. For this task, there are two useful libraries available in Python. 1. NLTK 2. Spacy. In this notebook, I will be working with Spacy because it is very fast and has many useful features compared to NLTK. So without further do let's get started!"},{"metadata":{"_kg_hide-output":false,"trusted":true,"_uuid":"25fd7f9ae5fa5ca3efe698e99ba8e1631f7121a6","_kg_hide-input":true},"cell_type":"code","source":"!python -m spacy download en_core_web_lg\nnlp = spacy.load('en_core_web_lg')\ndef normalize_text(text):\n    tm1 = re.sub('<pre>.*?</pre>', '', text, flags=re.DOTALL)\n    tm2 = re.sub('<code>.*?</code>', '', tm1, flags=re.DOTALL)\n    tm3 = re.sub('<[^>]+>©', '', tm1, flags=re.DOTALL)\n    return tm3.replace(\"\\n\", \"\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a69744aafbd96fad87b1433c0cd3d40fbd4f04d2","_kg_hide-input":true},"cell_type":"code","source":"# in this step we are going to remove code syntax from text \nreviews['description_Cleaned_1'] = reviews['description'].apply(normalize_text)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"d38a9fb0751391660bc33de4431ef2f2c521606b"},"cell_type":"code","source":"print('Before normalizing text-----\\n')\nprint(reviews['description'][2])\nprint('\\nAfter normalizing text-----\\n')\nprint(reviews['description_Cleaned_1'][2])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0875039d74a5dc0636bbfb2eb93efcf7278c1f47"},"cell_type":"markdown","source":"We can see a huge difference after normalizing our text. Now we can see our text is more manageable. This will help us to explore the reviews and later making summarizer.\n\nWe are also seeing that there are some punctuation and stopwords. We also don't need them. In the first place, I don't remove them because we are gonna need this in future when we will make summarizer. So let's make another column that will store our normalized text without punctuation and stopwords."},{"metadata":{"_uuid":"1d3e99e4511e5e57a41935fe1432d93a2745cfe1"},"cell_type":"markdown","source":"## 3.1 Clean text before feeding it to spaCy"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"64f118a88d5c7fb5fcf276f017fe6318b8b7c45e"},"cell_type":"code","source":"punctuations = '!\"#$%&\\'()*+,-/:;<=>?@[\\\\]^_`{|}~©'\n# Define function to cleanup text by removing personal pronouns, stopwords, and puncuation\ndef cleanup_text(docs, logging=False):\n    texts = []\n    doc = nlp(docs, disable=['parser', 'ner'])\n    tokens = [tok.lemma_.lower().strip() for tok in doc if tok.lemma_ != '-PRON-']\n    tokens = [tok for tok in tokens if tok not in stopwords and tok not in punctuations]\n    tokens = ' '.join(tokens)\n    texts.append(tokens)\n    return pd.Series(texts)\nreviews['Description_Cleaned'] = reviews['description_Cleaned_1'].apply(lambda x: cleanup_text(x, False))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e9d814a5a03b4063b9e67efc2c6cf330ec8fc443"},"cell_type":"code","source":"print('Reviews description with punctuatin and stopwords---\\n')\nprint(reviews['description_Cleaned_1'][0])\nprint('\\nReviews description after removing punctuation and stopwrods---\\n')\nprint(reviews['Description_Cleaned'][0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"05c4d720356799bcb929093c51fd9cb4af315317"},"cell_type":"markdown","source":"Wow! See! Now our text looks much readable and less messy!"},{"metadata":{"_uuid":"4a925e4a30c7ace8cb844ddbbadb0bd2a848f596"},"cell_type":"markdown","source":"# 4. Distribution of Points\nIn this section, I will try understand the distribution of points. Here points mean number of upvote the \tdescription got in social media(such as facebook,twitter etc)."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"e981d5761ded4aea90b302aa0cb0d6ac9ed7c27a"},"cell_type":"code","source":"plt.subplot(1, 2, 1)\n(reviews['points']).plot.hist(bins=30, figsize=(30,5), edgecolor='white',range=[0,150])\nplt.xlabel('Number of points', fontsize=17)\nplt.ylabel('frequency', fontsize=17)\nplt.tick_params(labelsize=15)\nplt.title('Number of points description', fontsize=17)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bb3caa428606c67c108d91c4ae7d4fa3f528d7a2"},"cell_type":"markdown","source":"The description of points lies between 80 to 100 mostly. Majority of the description got points between 80 to 100."},{"metadata":{"_uuid":"1642a971b3a627e10884a150714963ef2333e72b"},"cell_type":"markdown","source":"# 5. Analyze reviews description\nIn this section, I will try to analyze wine description. In Wine Reviews, the wine description plays a vital role. A good description can make your wine  stand out. It also helps get a reviews faster. Lastly, It will help you get some points. Let's see what we can find in the  wine description."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"e68dde499af6efa3a8a942b362b8b095a7fcace1"},"cell_type":"code","source":"reviews['Title_len'] = reviews['Description_Cleaned'].str.split().str.len()\nrev = reviews.groupby('Title_len')['points'].mean().reset_index()\ntrace1 = go.Scatter(\n    x = rev['Title_len'],\n    y = rev['points'],\n    mode = 'lines+markers',\n    name = 'lines+markers'\n)\nlayout = dict(title= 'Average points by wine description Length',\n              yaxis = dict(title='Average points'),\n              xaxis = dict(title='wine description Length'))\nfig=dict(data=[trace1], layout=layout)\npy.iplot(fig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"39abb8b4ce723d291e26f238ee6aad6a1e93ff27"},"cell_type":"markdown","source":"# 6. Description Summarizer\n![Imgur](https://i.imgur.com/DrvohGg.jpg?1)\n> In this step, I will try to make a description summarizer. There is a huge amount of research going for text summarization. But I will try to do a simple technique for text summarization. The technique describes below."},{"metadata":{"_uuid":"7f24b07ce1634f1a568f67bb6fd4a881c12fa113"},"cell_type":"markdown","source":"### 6.1 Convert Paragraphs to Sentences\n> We first need to convert the whole paragraph into sentences. The most common way of converting paragraphs to sentences is to split the paragraph whenever a period is encountered.\n\n### 6.2 Text Preprocessing\n> After converting paragraph to sentences, we need to remove all the special characters, stop words and numbers from all the sentences.\n\n### 6.3 Tokenizing the Sentences\n> We need to tokenize all the sentences to get all the words that exist in the sentences\n\n### 6.4 4. Find Weighted Frequency of Occurrence\n> Next we need to find the weighted frequency of occurrences of all the words. We can find the weighted frequency of each word by dividing its frequency by the frequency of the most occurring word.\n\n### 6.5 Replace Words by Weighted Frequency in Original Sentences\n> The final step is to plug the weighted frequency in place of the corresponding words in original sentences and finding their sum. It is important to mention that weighted frequency for the words removed during preprocessing (stop words, punctuation, digits etc.) will be zero and therefore is not required to be added\n\n### 6.6 Sort Sentences in Descending Order of Sum\n> The final step is to sort the sentences in inverse order of their sum. The sentences with highest frequencies summarize the text."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"231232c4be343eb6c6812ca26069bc437c422be8"},"cell_type":"code","source":"# this is function for text summarization\ndef generate_summary(text_without_removing_dot, cleaned_text):\n    sample_text = text_without_removing_dot\n    doc = nlp(sample_text)\n    sentence_list=[]\n    for idx, sentence in enumerate(doc.sents): # we are using spacy for sentence tokenization\n        sentence_list.append(re.sub(r'[^\\w\\s]','',str(sentence)))\n\n    stopwords = nltk.corpus.stopwords.words('english')\n\n    word_frequencies = {}  \n    for word in nltk.word_tokenize(cleaned_text):  \n        if word not in stopwords:\n            if word not in word_frequencies.keys():\n                word_frequencies[word] = 1\n            else:\n                word_frequencies[word] += 1\n\n\n    maximum_frequncy = max(word_frequencies.values())\n\n    for word in word_frequencies.keys():  \n        word_frequencies[word] = (word_frequencies[word]/maximum_frequncy)\n\n\n    sentence_scores = {}  \n    for sent in sentence_list:  \n        for word in nltk.word_tokenize(sent.lower()):\n            if word in word_frequencies.keys():\n                if len(sent.split(' ')) < 30:\n                    if sent not in sentence_scores.keys():\n                        sentence_scores[sent] = word_frequencies[word]\n                    else:\n                        sentence_scores[sent] += word_frequencies[word]\n\n\n    summary_sentences = heapq.nlargest(7, sentence_scores, key=sentence_scores.get)\n\n    summary = ' '.join(summary_sentences)\n    print(\"Original Text:\\n\")\n    print(text_without_removing_dot)\n    print('\\n\\nSummarized text:\\n')\n    print(summary)  ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ac545db73e4e8f7b7528a9743d163e78345d1130"},"cell_type":"markdown","source":"Now we have written the function let's try to summarize some descriptions."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"716d14346ed32a3530b28ff69e656b304344c054"},"cell_type":"code","source":"generate_summary(reviews['description_Cleaned_1'][8], reviews['Description_Cleaned'][8])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"6626cafc63770b6c170e956ec5bdbd0616424d7c"},"cell_type":"code","source":"generate_summary(reviews['description_Cleaned_1'][100], reviews['Description_Cleaned'][100])","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"f8b4740ed18aea47c70b34fcc993f518b443c330"},"cell_type":"code","source":"generate_summary(reviews['description_Cleaned_1'][500], reviews['Description_Cleaned'][500])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bd90fbd12a610786d89fe5d6af135dcad3d97132"},"cell_type":"markdown","source":"That's awesome! We successfully made a simple winemag description summarizer."},{"metadata":{"_uuid":"65c5b5c8280fdd4dc9b16fad86669cc444b5b02c"},"cell_type":"markdown","source":"# 7. Conclusion\n> Thanks for reading this notebook. If you have any suggestion feel free to reach me in the comment. And don't forget to upvote. 👍\n> Stay in touch for more update. Thank you. &#128526;"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}