Twiiter demographics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The head of marketing at a fashion company XYZ has a theory that male customers are more likely to engage with your product on social media than female customers and has asked you (a Data Scientist) to write an algorithm to predict the gender of Twitter users mentioning their product based on the text of their posts. The marketing head has provided you with a list of TweetIDs for each customer. You have to write a script that turns these lists of IDs into both a score representing how strongly the company believes them to be of a given gender and a prediction about their gender."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, the idea for you to get the tweets from the given IDs first. Then extract the text and apply sentiment analysis to predict their genders along with gender scores. This will enable to company to validate their hypothesis (not a statistical hypothesis though). \n",
    "\n",
    "I am not going to show any fancy sentiment analysis algorithm here. Instead I will create a small lexicon manually (I know this is terrible) and use it for making predictions. The objective here is to make use of parallel programming in real-life scenarios.\n",
    "\n",
    "If you are following along, you would need a **Twitter Developer Account**. Along with that the following Python libraries:\n",
    "- `python-twitter`\n",
    "- `toolz`\n",
    "- `multiprocessing`\n",
    "\n",
    "All of the above-mentioned libraries are pip installable. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import necessary packages\n",
    "import twitter\n",
    "from toolz import pipe\n",
    "from multiprocessing import Pool"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setup your Twitter API with your credentials\n",
    "Twitter = twitter.Api(consumer_key=\"\",\n",
    "            consumer_secret=\"\",\n",
    "            access_token_key=\"\",\n",
    "            access_token_secret=\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Helper functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Retrieve the tweets represented by given ids\n",
    "def get_tweet_from_id(tweet_id, api=Twitter):\n",
    "    return api.GetStatus(tweet_id, trim_user=True)\n",
    "\n",
    "# Extract the text from the tweets\n",
    "def tweet_to_text(tweet):\n",
    "    return tweet.text\n",
    "\n",
    "# Split the tweet text w.r.t whitespace. Don't bash! \n",
    "def tokenize_text(text):\n",
    "    return text.split()\n",
    "\n",
    "# Define the lexicon to score\n",
    "def score_text(tokens):\n",
    "    lexicon = {\"the\":1, \"to\":1, \"and\":1, \n",
    "    \"in\":1, \"have\":1, \"it\":1,\n",
    "    \"be\":-1, \"of\":-1, \"a\":-1, \n",
    "    \"that\":-1, \"i\":-1, \"for\":-1}\n",
    "    return sum(map(lambda x: lexicon.get(x, 0), tokens))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Construct a function pipeline\n",
    "def score_tweet(tweet_id):\n",
    "    return pipe(tweet_id, get_tweet_from_id, tweet_to_text,\n",
    "    tokenize_text, score_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from toolz import compose\n",
    "\n",
    "# Score the users based on their tweets\n",
    "def score_user(tweets): \n",
    "    N = len(tweets) \n",
    "    total = sum(map(score_tweet, tweets)) \n",
    "    return total/N \n",
    "\n",
    "# Get the gender\n",
    "def categorize_user(user_score): \n",
    "    if user_score > 0: \n",
    "        return {\"score\":user_score,\n",
    "                    \"gender\": \"Male\"}\n",
    "    return {\"score\":user_score, \n",
    "                \"gender\":\"Female\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## In action"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'score': -0.4, 'gender': 'Female'}, {'score': 0.0, 'gender': 'Female'}, {'score': 0.8, 'gender': 'Male'}, {'score': -0.4, 'gender': 'Female'}]\n"
     ]
    }
   ],
   "source": [
    "users_tweets = [\n",
    "    [1056365937547534341, 1056310126255034368, 1055985345341251584,\n",
    "    1056585873989394432, 1056585871623966720],\n",
    "    [1055986452612419584, 1056318330037002240, 1055957256162942977,\n",
    "     1056585921154420736, 1056585896898805766],\n",
    "    [1056240773572771841, 1056184836900175874, 1056367465477951490,\n",
    "     1056585972765224960, 1056585968155684864],\n",
    "    [1056452187897786368, 1056314736546115584, 1055172336062816258,\n",
    "     1056585983175602176, 1056585980881207297]]\n",
    "\n",
    "gender_pipeline = compose(categorize_user, score_user)\n",
    "\n",
    "with Pool() as P:\n",
    "    print(P.map(gender_pipeline, users_tweets))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}