Indeed_webscrape.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `Python Webscraping Indeed`\n",
    "\n",
    "# <font color=red>Mr Fugu Data Science</font>\n",
    "\n",
    "# (◕‿◕✿)\n",
    "\n",
    "# Purpose & Outcome:\n",
    "\n",
    "+ `Webscrape Indeed`: take position, job title,location,date of posting, string of qualifications\n",
    "\n",
    "+ Use a list of words related to your skills or skills you are interested in and extract from job post qualifications section.\n",
    "\n",
    "+ Date time formating\n",
    "\n",
    "`------------------------------`\n",
    "\n",
    "# `Next video Word Cloud and NLP using these data`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pip3 install wordcloud # (if you don't have already)\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import PIL              # for images\n",
    "import wordcloud        # ploting word cloud\n",
    "import requests         # grab web-page\n",
    "import pickle           # save file\n",
    "from bs4 import BeautifulSoup as bsopa  # parse web-page\n",
    "import datetime         # format date/time\n",
    "from collections import defaultdict\n",
    "import re               # regular expressions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# if you don't have wordcloud, get that now:\n",
    "\n",
    "# !pip3 install wordcloud\n",
    "# conda install -c conda-forge wordcloud"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "ex.) https://www.indeed.com/jobs?q=data+scientist&l=california&start=10\n",
    "\n",
    "range(0,150,10): each page will have \"start=0,start=10,start=20 etc\" which deals with\n",
    "going through each page, but not exactly 10 entries/pg. \n",
    "\n",
    "string formatting is used to denote what job and location we want: you can use a string\n",
    "separated by space and it will be interpreted by the website. \n",
    "\n",
    "sou=bsopa(y.text,'lxml') is taking our get request and converting to text in the format\n",
    "of 'lxml' but we can replace with 'html.parser' as well.\n",
    "\n",
    "Each job post can be parsed by the 'div',{\"class\":\"jobsearch-SerpJobCard\"} or \n",
    "'div', {'class': 'row'} depending on how you want to search\n",
    "\n",
    "After that we get each piece of information we want to obtain: job title, location etc.\n",
    "\n",
    "the only difficult and frustrating part is getting all the raw text for each posting\n",
    "relating to the qulifications. We have to open a link, iterate through it and then extract\n",
    "all the information with a try/except block and then further process.\n",
    "'''\n",
    "\n",
    "gg=[]\n",
    "for j in range(0,150,10):\n",
    "    position,location='data scientist','california'\n",
    "    y=requests.get('https://www.indeed.com/jobs?q={}&l={}&sort=date='.format(position,location)+str(j))\n",
    "\n",
    "    # y=requests.get('https://www.indeed.com/jobs?q=data+scientist&l=california&sort=date='+str(i))\n",
    "    sou=bsopa(y.text,'html.parser')\n",
    "\n",
    "#     for ii in sou.find_all('div', {'class': 'row'}):\n",
    "    for ii in sou.find_all('div',{\"class\":\"jobsearch-SerpJobCard\"}):\n",
    "\n",
    "        job_title = ii.find('a', {'data-tn-element': 'jobTitle'})['title']\n",
    "        company_name = ii.find('span', {'class': 'company'}).text.strip()    \n",
    "        location=ii.find('span',{\"class\":\"location\"})\n",
    "        post_date = ii.find('span', attrs={'class': 'date'})\n",
    "        summary=ii.find('div',attrs={'class':'summary'})\n",
    "\n",
    "        if location:\n",
    "            location=location.text.strip()\n",
    "        else:\n",
    "            location=ii.find('div',{\"class\":\"location\"})\n",
    "            location=location.text.strip()\n",
    "\n",
    "        k=ii.find('h2', {'class':\"title\"})\n",
    "        p=k.find(href=True)\n",
    "        v=p['href']\n",
    "        f_=str(v).replace('&amp;','&') # links to iterate for qualification text\n",
    "        \n",
    "        \n",
    "        datum = {'job_title': job_title,\n",
    "                    'company_name': company_name,\n",
    "                    'location': location,\n",
    "                    'summary':summary.text.strip(),\n",
    "                    'post_Date':post_date.text,\n",
    "                    'Qualification_link': f_}\n",
    "\n",
    "#         gg.append([location,job_title,company_name,post_date.text,summary.text.strip()\n",
    "#                   ,f_]) \n",
    "        gg.append(datum)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(gg)\n",
    "# gg[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `Links to extract qualification information`\n",
    "\n",
    "+ `This will be used to take the links used for when you click a job and it expands the small text`\n",
    "\n",
    "+ `After you exand the text it will show the full job description and relative information`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0,\n",
       " 'The CompanyThe world is facing a medical crisis, bacteria are increasingly evolving resistance to even our strongest antibiotics. The problem is already very real and immediate; for example, bloodstream infection leading to sepsis is now responsible for more than half of all deaths in hospitals and is the most expensive condition treated in hospitals. Sepsis mortality rate increases >6% every hour without effective antibiotic treatment. Yet, despite the life and death urgency, and healthcare cost impacts, current methodologies require 3 days to determine the correct antibiotic.']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get all qualification page text: key=index, value=string of text for qualification\n",
    "hoop=[]\n",
    "for i in range(len(gg)):\n",
    "    op=requests.get('https://www.indeed.com'+gg[i]['Qualification_link'])\n",
    "    sou_=bsopa(op.text,'html.parser')\n",
    "    for ii in sou_.find('div',{'class':'jobsearch-jobDescriptionText'}):\n",
    "        try:\n",
    "            hoop.append([i,''.join(ii.text.strip())])\n",
    "        except AttributeError:\n",
    "            hoop.append([i,''])\n",
    "hoop[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create dictionary with values as lists\n",
    "dct_lst= defaultdict(list)\n",
    "for i in hoop:\n",
    "    dct_lst[i[0]].append(i[1])\n",
    "u=[]\n",
    "for i in dct_lst.values(): # string join: lists of lists of strings\n",
    "    u.append(''.join(i))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Our Mission\\nWe’re here to create a safer, happier and more mindful future for all with the help of data science, engineering, design, and mobile technology. We're starting by reinventing insurance, by rethinking the technologies that enable it, but our true goal is to build a platform that rewards people for driving well — creating safer roads with fewer accidents in the process.\\n\\nBacked by impressive funding, we're poised to re-engineer a trillion-dollar category. We’re using rich customer insights, advanced technology and data science to build our cloud-native InsurTech solution. We're out to change behavior and promote mindful living.\\n\\nJob Requirements\\nYou must have all the skills and experiences associated with one of the options below. These options change over time and reflect the current needs of the team.\\nOption one - Telematics ML\\nMachine learning: Kaggle grandmaster-level skills with history of deploying models to production\\nTelematics experience: You have led telematics modeling at a major company known for telematics\\nSoftware development in Python: Strong scientific Python skills using pandas, scikit-learn, and statsmodels, as well as strong understanding of model deployment patterns\\nOption two - Telematics DL\\nDeep learning: Experience contributing new deep learning research to solve previously unsolved problems\\nTelematics experience: You have led telematics modeling at a major company known for telematics\\nSoftware development in Python: Strong scientific Python skills using pandas, scikit-learn, and statsmodels, as well as strong understanding of model deployment patterns\\nOption Three - Contact Center Optimization\\nContact center knowledge: You are an expert at using data science methods to optimize contact centers\\nMachine learning: Strong understanding of ML theory with significant applied experience\\nSoftware development in Python: Strong scientific Python skills using pandas, scikit-learn, and statsmodels, as well as strong understanding of model deployment patterns\\nMore details\\nSalary: We invest in first-rate people and pay top-of-market salaries for most positions, factoring in experience, talent and location. We do not offer equity.\\nBenefits: Medical, dental, vision, 401(k), wellness reimbursement, four weeks of vacation + six weeks of parental leave, and great work-life balance. Our office building offers on-site shower and bike stalls, and panoramic views of San Francisco.\\nLocation: Due to COVID-19 our teams are all working remotely through 2020. We provide an in-home office set-up including laptop, monitor, ergonomic desk, chair and other items as needed\\nLocation: Post COVID-19: San Francisco, CA near Montgomery Bart Station\\nAll are welcome at Blue Owl. We are an equal opportunity and affirmative action employer who values diversity and inclusion and looks for applicants who understand, embrace and thrive in a multicultural world. We do not discriminate on the basis of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity or Veteran status. Pursuant to the San Francisco Fair Chance Ordinance, we will consider for employment qualified applicants with arrest and conviction records.\\n\\n If you are a San Francisco resident, please read the City and County of San Francisco's Fair Chance Ordinance notice.\\nhttps://sfgov.org/olse/sites/default/files/FCO%20poster2020.pdf\""
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# one entry of our qualification text:\n",
    "u[2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>job_title</th>\n",
       "      <th>company_name</th>\n",
       "      <th>location</th>\n",
       "      <th>summary</th>\n",
       "      <th>post_Date</th>\n",
       "      <th>Qualification_link</th>\n",
       "      <th>Qual_Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Data Scientist, Medical Diagnostics</td>\n",
       "      <td>Specific Diagnostics</td>\n",
       "      <td>Mountain View, CA 94043</td>\n",
       "      <td>The development of data-driven visualization t...</td>\n",
       "      <td>30+ days ago</td>\n",
       "      <td>/pagead/clk?mo=r&amp;ad=-6NYlbfkN0ASXGwdLWjBNYivRa...</td>\n",
       "      <td>The CompanyThe world is facing a medical crisi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Data Scientist</td>\n",
       "      <td>Laxmi Therapeutic Devices</td>\n",
       "      <td>Goleta, CA 93117</td>\n",
       "      <td>7+ years' practical experience manipulating da...</td>\n",
       "      <td>16 days ago</td>\n",
       "      <td>/pagead/clk?mo=r&amp;ad=-6NYlbfkN0ALgD31io3l0I0Y-r...</td>\n",
       "      <td>Data ScientistLaxmi Therapeutic Devices – Gole...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Data Scientist</td>\n",
       "      <td>Blue Owl</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>We’re using rich customer insights, advanced t...</td>\n",
       "      <td>30+ days ago</td>\n",
       "      <td>/pagead/clk?mo=r&amp;ad=-6NYlbfkN0D3UvD5kBSgX9r9tF...</td>\n",
       "      <td>Our Mission\\nWe’re here to create a safer, hap...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Data Engineer</td>\n",
       "      <td>Amick Brown, LLC</td>\n",
       "      <td>Sunnyvale, CA</td>\n",
       "      <td>Develops technical tools and programming that ...</td>\n",
       "      <td>Today</td>\n",
       "      <td>/pagead/clk?mo=r&amp;ad=-6NYlbfkN0A74pTrSPrBtiJlYH...</td>\n",
       "      <td>Data EngineerSunnyvale, CAAmick Brown is seeki...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Data Scientist</td>\n",
       "      <td>Triplebyte</td>\n",
       "      <td>California</td>\n",
       "      <td>You'll report directly to Triplebytes' Head of...</td>\n",
       "      <td>22 days ago</td>\n",
       "      <td>/pagead/clk?mo=r&amp;ad=-6NYlbfkN0AMr11YIOo206dX9C...</td>\n",
       "      <td>About Triplebyte\\n\\nTriplebyte is transforming...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             job_title               company_name  \\\n",
       "0  Data Scientist, Medical Diagnostics       Specific Diagnostics   \n",
       "1                       Data Scientist  Laxmi Therapeutic Devices   \n",
       "2                       Data Scientist                   Blue Owl   \n",
       "3                        Data Engineer           Amick Brown, LLC   \n",
       "4                       Data Scientist                 Triplebyte   \n",
       "\n",
       "                  location                                            summary  \\\n",
       "0  Mountain View, CA 94043  The development of data-driven visualization t...   \n",
       "1         Goleta, CA 93117  7+ years' practical experience manipulating da...   \n",
       "2        San Francisco, CA  We’re using rich customer insights, advanced t...   \n",
       "3            Sunnyvale, CA  Develops technical tools and programming that ...   \n",
       "4               California  You'll report directly to Triplebytes' Head of...   \n",
       "\n",
       "      post_Date                                 Qualification_link  \\\n",
       "0  30+ days ago  /pagead/clk?mo=r&ad=-6NYlbfkN0ASXGwdLWjBNYivRa...   \n",
       "1   16 days ago  /pagead/clk?mo=r&ad=-6NYlbfkN0ALgD31io3l0I0Y-r...   \n",
       "2  30+ days ago  /pagead/clk?mo=r&ad=-6NYlbfkN0D3UvD5kBSgX9r9tF...   \n",
       "3         Today  /pagead/clk?mo=r&ad=-6NYlbfkN0A74pTrSPrBtiJlYH...   \n",
       "4   22 days ago  /pagead/clk?mo=r&ad=-6NYlbfkN0AMr11YIOo206dX9C...   \n",
       "\n",
       "                                           Qual_Text  \n",
       "0  The CompanyThe world is facing a medical crisi...  \n",
       "1  Data ScientistLaxmi Therapeutic Devices – Gole...  \n",
       "2  Our Mission\\nWe’re here to create a safer, hap...  \n",
       "3  Data EngineerSunnyvale, CAAmick Brown is seeki...  \n",
       "4  About Triplebyte\\n\\nTriplebyte is transforming...  "
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jobs_=pd.concat([pd.DataFrame(gg),pd.DataFrame(u,columns=['Qual_Text'])],axis=1)\n",
    "jobs_.head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `Parse posting date`: convert string data to official dates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['09-10-2020', '09-24-2020', '09-10-2020', '10-10-2020', '09-18-2020']"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import re\n",
    "v=[]\n",
    "for i in jobs_['post_Date']:\n",
    "\n",
    "    if re.findall(r'[0-9]',i):\n",
    "        # if the string has digits convert each entry to single string: ['3','0']->'30'\n",
    "        b=''.join(re.findall(r'[0-9]',i))\n",
    "        \n",
    "        # convert string int to int and subtract from today's date and format\n",
    "        g=(datetime.datetime.today()-datetime.timedelta(int(b))).strftime('%m-%d-%Y')\n",
    "\n",
    "        v.append(g)\n",
    "        \n",
    "    else: # this will contain strings like: 'just posted' or 'today' etc before convert\n",
    "        v.append(datetime.datetime.today().strftime('%m-%d-%Y'))\n",
    "v[:5]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>job_title</th>\n",
       "      <th>company_name</th>\n",
       "      <th>location</th>\n",
       "      <th>summary</th>\n",
       "      <th>post_Date</th>\n",
       "      <th>Qualification_link</th>\n",
       "      <th>Qual_Text</th>\n",
       "      <th>posting_date_fixed</th>\n",
       "      <th>skill_matches</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Data Scientist, Medical Diagnostics</td>\n",
       "      <td>Specific Diagnostics</td>\n",
       "      <td>Mountain View, CA 94043</td>\n",
       "      <td>The development of data-driven visualization t...</td>\n",
       "      <td>30+ days ago</td>\n",
       "      <td>/pagead/clk?mo=r&amp;ad=-6NYlbfkN0ASXGwdLWjBNYivRa...</td>\n",
       "      <td>The CompanyThe world is facing a medical crisi...</td>\n",
       "      <td>09-10-2020</td>\n",
       "      <td>[python, sql, visualization]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Data Scientist</td>\n",
       "      <td>Laxmi Therapeutic Devices</td>\n",
       "      <td>Goleta, CA 93117</td>\n",
       "      <td>7+ years' practical experience manipulating da...</td>\n",
       "      <td>16 days ago</td>\n",
       "      <td>/pagead/clk?mo=r&amp;ad=-6NYlbfkN0ALgD31io3l0I0Y-r...</td>\n",
       "      <td>Data ScientistLaxmi Therapeutic Devices – Gole...</td>\n",
       "      <td>09-24-2020</td>\n",
       "      <td>[python, sql, statistics, algorithms]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Data Scientist</td>\n",
       "      <td>Blue Owl</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>We’re using rich customer insights, advanced t...</td>\n",
       "      <td>30+ days ago</td>\n",
       "      <td>/pagead/clk?mo=r&amp;ad=-6NYlbfkN0D3UvD5kBSgX9r9tF...</td>\n",
       "      <td>Our Mission\\nWe’re here to create a safer, hap...</td>\n",
       "      <td>09-10-2020</td>\n",
       "      <td>[python, machine learning, deep learning, pandas]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Data Engineer</td>\n",
       "      <td>Amick Brown, LLC</td>\n",
       "      <td>Sunnyvale, CA</td>\n",
       "      <td>Develops technical tools and programming that ...</td>\n",
       "      <td>Today</td>\n",
       "      <td>/pagead/clk?mo=r&amp;ad=-6NYlbfkN0A74pTrSPrBtiJlYH...</td>\n",
       "      <td>Data EngineerSunnyvale, CAAmick Brown is seeki...</td>\n",
       "      <td>10-10-2020</td>\n",
       "      <td>[python, sql, aws, machine learning]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Data Scientist</td>\n",
       "      <td>Triplebyte</td>\n",
       "      <td>California</td>\n",
       "      <td>You'll report directly to Triplebytes' Head of...</td>\n",
       "      <td>22 days ago</td>\n",
       "      <td>/pagead/clk?mo=r&amp;ad=-6NYlbfkN0AMr11YIOo206dX9C...</td>\n",
       "      <td>About Triplebyte\\n\\nTriplebyte is transforming...</td>\n",
       "      <td>09-18-2020</td>\n",
       "      <td>[aws, machine learning]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             job_title               company_name  \\\n",
       "0  Data Scientist, Medical Diagnostics       Specific Diagnostics   \n",
       "1                       Data Scientist  Laxmi Therapeutic Devices   \n",
       "2                       Data Scientist                   Blue Owl   \n",
       "3                        Data Engineer           Amick Brown, LLC   \n",
       "4                       Data Scientist                 Triplebyte   \n",
       "\n",
       "                  location                                            summary  \\\n",
       "0  Mountain View, CA 94043  The development of data-driven visualization t...   \n",
       "1         Goleta, CA 93117  7+ years' practical experience manipulating da...   \n",
       "2        San Francisco, CA  We’re using rich customer insights, advanced t...   \n",
       "3            Sunnyvale, CA  Develops technical tools and programming that ...   \n",
       "4               California  You'll report directly to Triplebytes' Head of...   \n",
       "\n",
       "      post_Date                                 Qualification_link  \\\n",
       "0  30+ days ago  /pagead/clk?mo=r&ad=-6NYlbfkN0ASXGwdLWjBNYivRa...   \n",
       "1   16 days ago  /pagead/clk?mo=r&ad=-6NYlbfkN0ALgD31io3l0I0Y-r...   \n",
       "2  30+ days ago  /pagead/clk?mo=r&ad=-6NYlbfkN0D3UvD5kBSgX9r9tF...   \n",
       "3         Today  /pagead/clk?mo=r&ad=-6NYlbfkN0A74pTrSPrBtiJlYH...   \n",
       "4   22 days ago  /pagead/clk?mo=r&ad=-6NYlbfkN0AMr11YIOo206dX9C...   \n",
       "\n",
       "                                           Qual_Text posting_date_fixed  \\\n",
       "0  The CompanyThe world is facing a medical crisi...         09-10-2020   \n",
       "1  Data ScientistLaxmi Therapeutic Devices – Gole...         09-24-2020   \n",
       "2  Our Mission\\nWe’re here to create a safer, hap...         09-10-2020   \n",
       "3  Data EngineerSunnyvale, CAAmick Brown is seeki...         10-10-2020   \n",
       "4  About Triplebyte\\n\\nTriplebyte is transforming...         09-18-2020   \n",
       "\n",
       "                                       skill_matches  \n",
       "0                       [python, sql, visualization]  \n",
       "1              [python, sql, statistics, algorithms]  \n",
       "2  [python, machine learning, deep learning, pandas]  \n",
       "3               [python, sql, aws, machine learning]  \n",
       "4                            [aws, machine learning]  "
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# fixed posting date to date format instead of string: last column\n",
    "jobs_['posting_date_fixed']=v\n",
    "jobs_.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "# `Create a list of skills that you may have or general list`\n",
    "\n",
    "+ This can be used in a few ways, for example if I put all my skills and see what matched up for jobs.\n",
    "\n",
    "+ We could generate a corpus and start parsing through job postings not just data scientist, but similar jobs as well. After all the matches for indeed are not the best!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'jobs_' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[1;32mIn[2], line 13\u001b[0m\n\u001b[0;32m     10\u001b[0m buzz_words_list\u001b[39m=\u001b[39m[x\u001b[39m.\u001b[39mlower() \u001b[39mfor\u001b[39;00m x \u001b[39min\u001b[39;00m buzz_words]  \u001b[39m# convert list to lowercase to parse\u001b[39;00m\n\u001b[0;32m     12\u001b[0m yo\u001b[39m=\u001b[39m[]\n\u001b[1;32m---> 13\u001b[0m \u001b[39mfor\u001b[39;00m i \u001b[39min\u001b[39;00m \u001b[39mrange\u001b[39m(\u001b[39mlen\u001b[39m(jobs_\u001b[39m.\u001b[39mQual_Text)):\n\u001b[0;32m     14\u001b[0m     a\u001b[39m=\u001b[39mbuzz_words_list\n\u001b[0;32m     15\u001b[0m     dd\u001b[39m=\u001b[39m[x \u001b[39mfor\u001b[39;00m x \u001b[39min\u001b[39;00m a \u001b[39mif\u001b[39;00m x \u001b[39min\u001b[39;00m jobs_\u001b[39m.\u001b[39mQual_Text[i]\u001b[39m.\u001b[39mlower()]\n",
      "\u001b[1;31mNameError\u001b[0m: name 'jobs_' is not defined"
     ]
    }
   ],
   "source": [
    "buzz_words=['Python','SQL','AWS', 'Machine learning','Deep learning','Text mining',\n",
    "'NLP','SAS','Tableau','Sagemaker','Tensorflow','Spark', 'numpy', 'MongDB','PSQL',\n",
    "\"Postgres\", 'Pandas', 'RESTFUL','NLP','Statistics','Algorithms','Visualization',\n",
    "'GCP','Google Cloud','Naive Bayes','Random Forest','Bachelors degree','Masters degree'\n",
    "'Java','Pyspark','Postgres','MySQL','Github','Docker','Machine Learning','C+',\n",
    "'C++','Pytorch','Jupyter Notebook','R Studio','R-Studio','Forecasting','Hive',\n",
    "'PhD','GCP','Numpy','NoSQL','Neo4j','Neural Network','Clustering','Linear Algebra',\n",
    "'Google Colab','Data Mining','Regression','Time Series','ETL','Data Wrangling',\n",
    "'Web Scraping','Feature Extraction','Featuring Engineering','Scipy','ML','DL']\n",
    "buzz_words_list=[x.lower() for x in buzz_words]  # convert list to lowercase to parse\n",
    "\n",
    "yo=[]\n",
    "for i in range(len(jobs_.Qual_Text)):\n",
    "    a=buzz_words_list\n",
    "    dd=[x for x in a if x in jobs_.Qual_Text[i].lower()]\n",
    "    yo.append(dd)\n",
    "jobs_['skill_matches']=yo\n",
    "jobs_.head(7)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>job_title</th>\n",
       "      <th>company_name</th>\n",
       "      <th>location</th>\n",
       "      <th>summary</th>\n",
       "      <th>post_Date</th>\n",
       "      <th>Qualification_link</th>\n",
       "      <th>Qual_Text</th>\n",
       "      <th>skill_matches</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Data Scientist, Medical Diagnostics</td>\n",
       "      <td>Specific Diagnostics</td>\n",
       "      <td>Mountain View, CA 94043</td>\n",
       "      <td>The development of data-driven visualization t...</td>\n",
       "      <td>30+ days ago</td>\n",
       "      <td>/pagead/clk?mo=r&amp;ad=-6NYlbfkN0ASXGwdLWjBNYivRa...</td>\n",
       "      <td>The CompanyThe world is facing a medical crisi...</td>\n",
       "      <td>[python, sql, visualization, c+, c++, nosql]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Data Scientist</td>\n",
       "      <td>Laxmi Therapeutic Devices</td>\n",
       "      <td>Goleta, CA 93117</td>\n",
       "      <td>7+ years' practical experience manipulating da...</td>\n",
       "      <td>16 days ago</td>\n",
       "      <td>/pagead/clk?mo=r&amp;ad=-6NYlbfkN0ALgD31io3l0I0Y-r...</td>\n",
       "      <td>Data ScientistLaxmi Therapeutic Devices – Gole...</td>\n",
       "      <td>[python, sql, statistics, algorithms]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Data Scientist</td>\n",
       "      <td>Blue Owl</td>\n",
       "      <td>San Francisco, CA</td>\n",
       "      <td>We’re using rich customer insights, advanced t...</td>\n",
       "      <td>30+ days ago</td>\n",
       "      <td>/pagead/clk?mo=r&amp;ad=-6NYlbfkN0D3UvD5kBSgX9r9tF...</td>\n",
       "      <td>Our Mission\\nWe’re here to create a safer, hap...</td>\n",
       "      <td>[python, machine learning, deep learning, pand...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Data Engineer</td>\n",
       "      <td>Amick Brown, LLC</td>\n",
       "      <td>Sunnyvale, CA</td>\n",
       "      <td>Develops technical tools and programming that ...</td>\n",
       "      <td>Today</td>\n",
       "      <td>/pagead/clk?mo=r&amp;ad=-6NYlbfkN0A74pTrSPrBtiJlYH...</td>\n",
       "      <td>Data EngineerSunnyvale, CAAmick Brown is seeki...</td>\n",
       "      <td>[python, sql, aws, machine learning, machine l...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Data Scientist</td>\n",
       "      <td>Triplebyte</td>\n",
       "      <td>California</td>\n",
       "      <td>You'll report directly to Triplebytes' Head of...</td>\n",
       "      <td>22 days ago</td>\n",
       "      <td>/pagead/clk?mo=r&amp;ad=-6NYlbfkN0AMr11YIOo206dX9C...</td>\n",
       "      <td>About Triplebyte\\n\\nTriplebyte is transforming...</td>\n",
       "      <td>[aws, machine learning, machine learning, time...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Data Scientist, Analytics</td>\n",
       "      <td>Evernote</td>\n",
       "      <td>Redwood City, CA 94063 (Downtown area)</td>\n",
       "      <td>Apply your expertise in quantitative analysis,...</td>\n",
       "      <td>3 days ago</td>\n",
       "      <td>/rc/clk?jk=e6e04f20bd7ac8f1&amp;fccid=4d2449c755ba...</td>\n",
       "      <td>About the team:\\n\\nThe Evernote Analytics team...</td>\n",
       "      <td>[python, sql, sas, tableau, statistics, visual...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Data Scientist</td>\n",
       "      <td>Apple</td>\n",
       "      <td>Santa Clara Valley, CA 95014</td>\n",
       "      <td>In-depth knowledge of digital analytics data, ...</td>\n",
       "      <td>Today</td>\n",
       "      <td>/rc/clk?jk=5a488b6a19d296c3&amp;fccid=c1099851e979...</td>\n",
       "      <td>Summary\\nPosted: Oct 9, 2020\\nWeekly Hours: 40...</td>\n",
       "      <td>[sql, tableau, visualization]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Data Scientist</td>\n",
       "      <td>RMDS Lab, Inc</td>\n",
       "      <td>South Lake, CA</td>\n",
       "      <td>Provide consultation on data science and machi...</td>\n",
       "      <td>17 days ago</td>\n",
       "      <td>/company/RMDS-Lab/jobs/Data-Scientist-c487fd99...</td>\n",
       "      <td>About RMDS LabRMDS Lab and its Global Associat...</td>\n",
       "      <td>[python, machine learning, machine learning]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Data Scientist 2</td>\n",
       "      <td>Xoom</td>\n",
       "      <td>San Francisco, CA 94105 (Financial District area)</td>\n",
       "      <td>Clear subject matter expert who can help mento...</td>\n",
       "      <td>Today</td>\n",
       "      <td>/rc/clk?jk=00c2ef3c615076e9&amp;fccid=978d9fd9799d...</td>\n",
       "      <td>Who we are: Fueled by a fundamental belief tha...</td>\n",
       "      <td>[statistics]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Data Science Intern</td>\n",
       "      <td>Parsec Education</td>\n",
       "      <td>Fresno, CA 93721 (Central area)</td>\n",
       "      <td>Parsec Education specializes in analyzing stat...</td>\n",
       "      <td>10 days ago</td>\n",
       "      <td>/pagead/clk?mo=r&amp;ad=-6NYlbfkN0B5bikR7eyU0Vk4cd...</td>\n",
       "      <td>Greetings Applicant!Here at Parsec Education w...</td>\n",
       "      <td>[python, sql, machine learning, visualization,...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             job_title               company_name  \\\n",
       "0  Data Scientist, Medical Diagnostics       Specific Diagnostics   \n",
       "1                       Data Scientist  Laxmi Therapeutic Devices   \n",
       "2                       Data Scientist                   Blue Owl   \n",
       "3                        Data Engineer           Amick Brown, LLC   \n",
       "4                       Data Scientist                 Triplebyte   \n",
       "5            Data Scientist, Analytics                   Evernote   \n",
       "6                       Data Scientist                      Apple   \n",
       "7                       Data Scientist              RMDS Lab, Inc   \n",
       "8                     Data Scientist 2                       Xoom   \n",
       "9                  Data Science Intern           Parsec Education   \n",
       "\n",
       "                                            location  \\\n",
       "0                            Mountain View, CA 94043   \n",
       "1                                   Goleta, CA 93117   \n",
       "2                                  San Francisco, CA   \n",
       "3                                      Sunnyvale, CA   \n",
       "4                                         California   \n",
       "5             Redwood City, CA 94063 (Downtown area)   \n",
       "6                       Santa Clara Valley, CA 95014   \n",
       "7                                     South Lake, CA   \n",
       "8  San Francisco, CA 94105 (Financial District area)   \n",
       "9                    Fresno, CA 93721 (Central area)   \n",
       "\n",
       "                                             summary     post_Date  \\\n",
       "0  The development of data-driven visualization t...  30+ days ago   \n",
       "1  7+ years' practical experience manipulating da...   16 days ago   \n",
       "2  We’re using rich customer insights, advanced t...  30+ days ago   \n",
       "3  Develops technical tools and programming that ...         Today   \n",
       "4  You'll report directly to Triplebytes' Head of...   22 days ago   \n",
       "5  Apply your expertise in quantitative analysis,...    3 days ago   \n",
       "6  In-depth knowledge of digital analytics data, ...         Today   \n",
       "7  Provide consultation on data science and machi...   17 days ago   \n",
       "8  Clear subject matter expert who can help mento...         Today   \n",
       "9  Parsec Education specializes in analyzing stat...   10 days ago   \n",
       "\n",
       "                                  Qualification_link  \\\n",
       "0  /pagead/clk?mo=r&ad=-6NYlbfkN0ASXGwdLWjBNYivRa...   \n",
       "1  /pagead/clk?mo=r&ad=-6NYlbfkN0ALgD31io3l0I0Y-r...   \n",
       "2  /pagead/clk?mo=r&ad=-6NYlbfkN0D3UvD5kBSgX9r9tF...   \n",
       "3  /pagead/clk?mo=r&ad=-6NYlbfkN0A74pTrSPrBtiJlYH...   \n",
       "4  /pagead/clk?mo=r&ad=-6NYlbfkN0AMr11YIOo206dX9C...   \n",
       "5  /rc/clk?jk=e6e04f20bd7ac8f1&fccid=4d2449c755ba...   \n",
       "6  /rc/clk?jk=5a488b6a19d296c3&fccid=c1099851e979...   \n",
       "7  /company/RMDS-Lab/jobs/Data-Scientist-c487fd99...   \n",
       "8  /rc/clk?jk=00c2ef3c615076e9&fccid=978d9fd9799d...   \n",
       "9  /pagead/clk?mo=r&ad=-6NYlbfkN0B5bikR7eyU0Vk4cd...   \n",
       "\n",
       "                                           Qual_Text  \\\n",
       "0  The CompanyThe world is facing a medical crisi...   \n",
       "1  Data ScientistLaxmi Therapeutic Devices – Gole...   \n",
       "2  Our Mission\\nWe’re here to create a safer, hap...   \n",
       "3  Data EngineerSunnyvale, CAAmick Brown is seeki...   \n",
       "4  About Triplebyte\\n\\nTriplebyte is transforming...   \n",
       "5  About the team:\\n\\nThe Evernote Analytics team...   \n",
       "6  Summary\\nPosted: Oct 9, 2020\\nWeekly Hours: 40...   \n",
       "7  About RMDS LabRMDS Lab and its Global Associat...   \n",
       "8  Who we are: Fueled by a fundamental belief tha...   \n",
       "9  Greetings Applicant!Here at Parsec Education w...   \n",
       "\n",
       "                                       skill_matches  \n",
       "0       [python, sql, visualization, c+, c++, nosql]  \n",
       "1              [python, sql, statistics, algorithms]  \n",
       "2  [python, machine learning, deep learning, pand...  \n",
       "3  [python, sql, aws, machine learning, machine l...  \n",
       "4  [aws, machine learning, machine learning, time...  \n",
       "5  [python, sql, sas, tableau, statistics, visual...  \n",
       "6                      [sql, tableau, visualization]  \n",
       "7       [python, machine learning, machine learning]  \n",
       "8                                       [statistics]  \n",
       "9  [python, sql, machine learning, visualization,...  "
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "filename='Indeed_scrape_Oct2020.txt'\n",
    "file=open(filename,'wb')\n",
    "pickle.dump(jobs_,file)\n",
    "\n",
    "file_ =open(filename,'rb')\n",
    "new_file_ =pickle.load(file_)\n",
    "new_file_.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <font color=red>LIKE</font>, Share &\n",
    "\n",
    "# <font color=red>SUB</font>scribe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `Next Video: Word Cloud From Indeed Job Postings, with NLP`\n",
    "\n",
    "`--------------------`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://www.jobspikr.com/blog/scraping-indeed-job-data-using-python/\n",
    "\n",
    "https://jlgamez.com/how-i-scrape-jobs-data-from-indeed-com-with-python/\n",
    "\n",
    "http://rstudio-pubs-static.s3.amazonaws.com/495949_1ab68b7e55cf4a3ab1fc00285e28fbcc.html\n",
    "\n",
    "https://chrislovejoy.me/job-scraper/\n",
    "\n",
    "https://www.youtube.com/watch?v=eN_3d4JrL_w"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.17"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}