04-Filtering - Using Conditionals to Filter.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 04 - Filtering - Using Conditionals to Filter Rows and Columns \n",
    "\n",
    "https://youtu.be/Lw2rlcxScZY?si=GV9Pm1IzeNSndh-p\n",
    "\n",
    "Notes by [Innovinitylabs](https://github.com/innovinitylabs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "people = {\n",
    "    \"first\": [\"Corey\", 'Jane', 'John'], \n",
    "    \"last\": [\"Schafer\", 'Doe', 'Doe'], \n",
    "    \"email\": [\"CoreyMSchafer@gmail.com\", 'JaneDoe@email.com', 'JohnDoe@email.com']\n",
    "}\n",
    "dft = pd.DataFrame(people)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to get everyone with last name Doe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we get only a `series` with yes or no"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    False\n",
       "1     True\n",
       "2     True\n",
       "Name: last, dtype: bool"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft['last'] == 'Doe'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "so we are adding it to a variable for clarity and then later we can use it to filter and get the data we need"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "filt = (dft['last'] == 'Doe') # () added for clarity\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    False\n",
       "1     True\n",
       "2     True\n",
       "Name: last, dtype: bool"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we can put it directly in the brackets for dataframe  but for clarity we are adding it variable and use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>email</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jane</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JaneDoe@email.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>John</td>\n",
       "      <td>Doe</td>\n",
       "      <td>JohnDoe@email.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  first last              email\n",
       "1  Jane  Doe  JaneDoe@email.com\n",
       "2  John  Doe  JohnDoe@email.com"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft[filt]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "but its better to use `.loc` since we can also filter using column if needed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    JaneDoe@email.com\n",
       "2    JohnDoe@email.com\n",
       "Name: email, dtype: object"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.loc[filt, 'email']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### AND OR Operator in pandas\n",
    "> AND = `&`  \n",
    "> OR  = `|`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "filt2 = (dft['last'] == 'Doe') & (dft['first'] == 'John')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2    JohnDoe@email.com\n",
       "Name: email, dtype: object"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.loc[filt2, 'email']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to negate the filter we use `~` (kinda like NOT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    CoreyMSchafer@gmail.com\n",
       "1          JaneDoe@email.com\n",
       "Name: email, dtype: object"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dft.loc[~filt2, 'email']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Survey Data "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('data/survey_results_public.csv', index_col='Respondent' )\n",
    "schema_df = pd.read_csv('data/survey_results_schema.csv', index_col='Column')\n",
    "pd.set_option('display.max_columns', 85)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pd.set_option('display.max_rows', 85)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MainBranch</th>\n",
       "      <th>Hobbyist</th>\n",
       "      <th>OpenSourcer</th>\n",
       "      <th>OpenSource</th>\n",
       "      <th>Employment</th>\n",
       "      <th>Country</th>\n",
       "      <th>Student</th>\n",
       "      <th>EdLevel</th>\n",
       "      <th>UndergradMajor</th>\n",
       "      <th>EduOther</th>\n",
       "      <th>OrgSize</th>\n",
       "      <th>DevType</th>\n",
       "      <th>YearsCode</th>\n",
       "      <th>Age1stCode</th>\n",
       "      <th>YearsCodePro</th>\n",
       "      <th>CareerSat</th>\n",
       "      <th>JobSat</th>\n",
       "      <th>MgrIdiot</th>\n",
       "      <th>MgrMoney</th>\n",
       "      <th>MgrWant</th>\n",
       "      <th>JobSeek</th>\n",
       "      <th>LastHireDate</th>\n",
       "      <th>LastInt</th>\n",
       "      <th>FizzBuzz</th>\n",
       "      <th>JobFactors</th>\n",
       "      <th>ResumeUpdate</th>\n",
       "      <th>CurrencySymbol</th>\n",
       "      <th>CurrencyDesc</th>\n",
       "      <th>CompTotal</th>\n",
       "      <th>CompFreq</th>\n",
       "      <th>ConvertedComp</th>\n",
       "      <th>WorkWeekHrs</th>\n",
       "      <th>WorkPlan</th>\n",
       "      <th>WorkChallenge</th>\n",
       "      <th>WorkRemote</th>\n",
       "      <th>WorkLoc</th>\n",
       "      <th>ImpSyn</th>\n",
       "      <th>CodeRev</th>\n",
       "      <th>CodeRevHrs</th>\n",
       "      <th>UnitTests</th>\n",
       "      <th>PurchaseHow</th>\n",
       "      <th>PurchaseWhat</th>\n",
       "      <th>LanguageWorkedWith</th>\n",
       "      <th>LanguageDesireNextYear</th>\n",
       "      <th>DatabaseWorkedWith</th>\n",
       "      <th>DatabaseDesireNextYear</th>\n",
       "      <th>PlatformWorkedWith</th>\n",
       "      <th>PlatformDesireNextYear</th>\n",
       "      <th>WebFrameWorkedWith</th>\n",
       "      <th>WebFrameDesireNextYear</th>\n",
       "      <th>MiscTechWorkedWith</th>\n",
       "      <th>MiscTechDesireNextYear</th>\n",
       "      <th>DevEnviron</th>\n",
       "      <th>OpSys</th>\n",
       "      <th>Containers</th>\n",
       "      <th>BlockchainOrg</th>\n",
       "      <th>BlockchainIs</th>\n",
       "      <th>BetterLife</th>\n",
       "      <th>ITperson</th>\n",
       "      <th>OffOn</th>\n",
       "      <th>SocialMedia</th>\n",
       "      <th>Extraversion</th>\n",
       "      <th>ScreenName</th>\n",
       "      <th>SOVisit1st</th>\n",
       "      <th>SOVisitFreq</th>\n",
       "      <th>SOVisitTo</th>\n",
       "      <th>SOFindAnswer</th>\n",
       "      <th>SOTimeSaved</th>\n",
       "      <th>SOHowMuchTime</th>\n",
       "      <th>SOAccount</th>\n",
       "      <th>SOPartFreq</th>\n",
       "      <th>SOJobs</th>\n",
       "      <th>EntTeams</th>\n",
       "      <th>SOComm</th>\n",
       "      <th>WelcomeChange</th>\n",
       "      <th>SONewContent</th>\n",
       "      <th>Age</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Trans</th>\n",
       "      <th>Sexuality</th>\n",
       "      <th>Ethnicity</th>\n",
       "      <th>Dependents</th>\n",
       "      <th>SurveyLength</th>\n",
       "      <th>SurveyEase</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Respondent</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>I am a student who is learning to code</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Never</td>\n",
       "      <td>The quality of OSS and closed source software ...</td>\n",
       "      <td>Not employed, and not looking for work</td>\n",
       "      <td>United Kingdom</td>\n",
       "      <td>No</td>\n",
       "      <td>Primary/elementary school</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Taught yourself a new language, framework, or ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>HTML/CSS;Java;JavaScript;Python</td>\n",
       "      <td>C;C++;C#;Go;HTML/CSS;Java;JavaScript;Python;SQL</td>\n",
       "      <td>SQLite</td>\n",
       "      <td>MySQL</td>\n",
       "      <td>MacOS;Windows</td>\n",
       "      <td>Android;Arduino;Windows</td>\n",
       "      <td>Django;Flask</td>\n",
       "      <td>Flask;jQuery</td>\n",
       "      <td>Node.js</td>\n",
       "      <td>Node.js</td>\n",
       "      <td>IntelliJ;Notepad++;PyCharm</td>\n",
       "      <td>Windows</td>\n",
       "      <td>I do not use containers</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Fortunately, someone else has that title</td>\n",
       "      <td>Yes</td>\n",
       "      <td>Twitter</td>\n",
       "      <td>Online</td>\n",
       "      <td>Username</td>\n",
       "      <td>2017</td>\n",
       "      <td>A few times per month or weekly</td>\n",
       "      <td>Find answers to specific questions;Learn how t...</td>\n",
       "      <td>3-5 times per week</td>\n",
       "      <td>Stack Overflow was much faster</td>\n",
       "      <td>31-60 minutes</td>\n",
       "      <td>No</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No, I didn't know that Stack Overflow had a jo...</td>\n",
       "      <td>No, and I don't know what those are</td>\n",
       "      <td>Neutral</td>\n",
       "      <td>Just as welcome now as I felt last year</td>\n",
       "      <td>Tech articles written by other developers;Indu...</td>\n",
       "      <td>14.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>No</td>\n",
       "      <td>Straight / Heterosexual</td>\n",
       "      <td>NaN</td>\n",
       "      <td>No</td>\n",
       "      <td>Appropriate in length</td>\n",
       "      <td>Neither easy nor difficult</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        MainBranch Hobbyist OpenSourcer  \\\n",
       "Respondent                                                                \n",
       "1           I am a student who is learning to code      Yes       Never   \n",
       "\n",
       "                                                   OpenSource  \\\n",
       "Respondent                                                      \n",
       "1           The quality of OSS and closed source software ...   \n",
       "\n",
       "                                        Employment         Country Student  \\\n",
       "Respondent                                                                   \n",
       "1           Not employed, and not looking for work  United Kingdom      No   \n",
       "\n",
       "                              EdLevel UndergradMajor  \\\n",
       "Respondent                                             \n",
       "1           Primary/elementary school            NaN   \n",
       "\n",
       "                                                     EduOther OrgSize DevType  \\\n",
       "Respondent                                                                      \n",
       "1           Taught yourself a new language, framework, or ...     NaN     NaN   \n",
       "\n",
       "           YearsCode Age1stCode YearsCodePro CareerSat JobSat MgrIdiot  \\\n",
       "Respondent                                                               \n",
       "1                  4         10          NaN       NaN    NaN      NaN   \n",
       "\n",
       "           MgrMoney MgrWant JobSeek LastHireDate LastInt FizzBuzz JobFactors  \\\n",
       "Respondent                                                                     \n",
       "1               NaN     NaN     NaN          NaN     NaN      NaN        NaN   \n",
       "\n",
       "           ResumeUpdate CurrencySymbol CurrencyDesc  CompTotal CompFreq  \\\n",
       "Respondent                                                                \n",
       "1                   NaN            NaN          NaN        NaN      NaN   \n",
       "\n",
       "            ConvertedComp  WorkWeekHrs WorkPlan WorkChallenge WorkRemote  \\\n",
       "Respondent                                                                 \n",
       "1                     NaN          NaN      NaN           NaN        NaN   \n",
       "\n",
       "           WorkLoc ImpSyn CodeRev  CodeRevHrs UnitTests PurchaseHow  \\\n",
       "Respondent                                                            \n",
       "1              NaN    NaN     NaN         NaN       NaN         NaN   \n",
       "\n",
       "           PurchaseWhat               LanguageWorkedWith  \\\n",
       "Respondent                                                 \n",
       "1                   NaN  HTML/CSS;Java;JavaScript;Python   \n",
       "\n",
       "                                     LanguageDesireNextYear  \\\n",
       "Respondent                                                    \n",
       "1           C;C++;C#;Go;HTML/CSS;Java;JavaScript;Python;SQL   \n",
       "\n",
       "           DatabaseWorkedWith DatabaseDesireNextYear PlatformWorkedWith  \\\n",
       "Respondent                                                                \n",
       "1                      SQLite                  MySQL      MacOS;Windows   \n",
       "\n",
       "             PlatformDesireNextYear WebFrameWorkedWith WebFrameDesireNextYear  \\\n",
       "Respondent                                                                      \n",
       "1           Android;Arduino;Windows       Django;Flask           Flask;jQuery   \n",
       "\n",
       "           MiscTechWorkedWith MiscTechDesireNextYear  \\\n",
       "Respondent                                             \n",
       "1                     Node.js                Node.js   \n",
       "\n",
       "                            DevEnviron    OpSys               Containers  \\\n",
       "Respondent                                                                 \n",
       "1           IntelliJ;Notepad++;PyCharm  Windows  I do not use containers   \n",
       "\n",
       "           BlockchainOrg BlockchainIs BetterLife  \\\n",
       "Respondent                                         \n",
       "1                    NaN          NaN        Yes   \n",
       "\n",
       "                                            ITperson OffOn SocialMedia  \\\n",
       "Respondent                                                               \n",
       "1           Fortunately, someone else has that title   Yes     Twitter   \n",
       "\n",
       "           Extraversion ScreenName SOVisit1st  \\\n",
       "Respondent                                      \n",
       "1                Online   Username       2017   \n",
       "\n",
       "                                SOVisitFreq  \\\n",
       "Respondent                                    \n",
       "1           A few times per month or weekly   \n",
       "\n",
       "                                                    SOVisitTo  \\\n",
       "Respondent                                                      \n",
       "1           Find answers to specific questions;Learn how t...   \n",
       "\n",
       "                  SOFindAnswer                     SOTimeSaved  SOHowMuchTime  \\\n",
       "Respondent                                                                      \n",
       "1           3-5 times per week  Stack Overflow was much faster  31-60 minutes   \n",
       "\n",
       "           SOAccount SOPartFreq  \\\n",
       "Respondent                        \n",
       "1                 No        NaN   \n",
       "\n",
       "                                                       SOJobs  \\\n",
       "Respondent                                                      \n",
       "1           No, I didn't know that Stack Overflow had a jo...   \n",
       "\n",
       "                                       EntTeams   SOComm  \\\n",
       "Respondent                                                 \n",
       "1           No, and I don't know what those are  Neutral   \n",
       "\n",
       "                                      WelcomeChange  \\\n",
       "Respondent                                            \n",
       "1           Just as welcome now as I felt last year   \n",
       "\n",
       "                                                 SONewContent   Age Gender  \\\n",
       "Respondent                                                                   \n",
       "1           Tech articles written by other developers;Indu...  14.0    Man   \n",
       "\n",
       "           Trans                Sexuality Ethnicity Dependents  \\\n",
       "Respondent                                                       \n",
       "1             No  Straight / Heterosexual       NaN         No   \n",
       "\n",
       "                     SurveyLength                  SurveyEase  \n",
       "Respondent                                                     \n",
       "1           Appropriate in length  Neither easy nor difficult  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>QuestionText</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Column</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Respondent</th>\n",
       "      <td>Randomized respondent ID number (not in order ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MainBranch</th>\n",
       "      <td>Which of the following options best describes ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Hobbyist</th>\n",
       "      <td>Do you code as a hobby?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>OpenSourcer</th>\n",
       "      <td>How often do you contribute to open source?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>OpenSource</th>\n",
       "      <td>How do you feel about the quality of open sour...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sexuality</th>\n",
       "      <td>Which of the following do you currently identi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ethnicity</th>\n",
       "      <td>Which of the following do you identify as? Ple...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Dependents</th>\n",
       "      <td>Do you have any dependents (e.g., children, el...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SurveyLength</th>\n",
       "      <td>How do you feel about the length of the survey...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SurveyEase</th>\n",
       "      <td>How easy or difficult was this survey to compl...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>85 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   QuestionText\n",
       "Column                                                         \n",
       "Respondent    Randomized respondent ID number (not in order ...\n",
       "MainBranch    Which of the following options best describes ...\n",
       "Hobbyist                                Do you code as a hobby?\n",
       "OpenSourcer         How often do you contribute to open source?\n",
       "OpenSource    How do you feel about the quality of open sour...\n",
       "...                                                         ...\n",
       "Sexuality     Which of the following do you currently identi...\n",
       "Ethnicity     Which of the following do you identify as? Ple...\n",
       "Dependents    Do you have any dependents (e.g., children, el...\n",
       "SurveyLength  How do you feel about the length of the survey...\n",
       "SurveyEase    How easy or difficult was this survey to compl...\n",
       "\n",
       "[85 rows x 1 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "schema_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To filter high salary and know what languages they know"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "filt_highsal = (df['ConvertedComp'] > 70000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Country</th>\n",
       "      <th>LanguageWorkedWith</th>\n",
       "      <th>ConvertedComp</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Respondent</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Canada</td>\n",
       "      <td>Java;R;SQL</td>\n",
       "      <td>366420.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>New Zealand</td>\n",
       "      <td>Bash/Shell/PowerShell;C#;HTML/CSS;JavaScript;P...</td>\n",
       "      <td>95179.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>United States</td>\n",
       "      <td>Bash/Shell/PowerShell;HTML/CSS;JavaScript;PHP;...</td>\n",
       "      <td>90000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>United Kingdom</td>\n",
       "      <td>Bash/Shell/PowerShell;C#;HTML/CSS;JavaScript;T...</td>\n",
       "      <td>455352.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>United States</td>\n",
       "      <td>Bash/Shell/PowerShell;C++;HTML/CSS;JavaScript;...</td>\n",
       "      <td>103000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88876</th>\n",
       "      <td>United States</td>\n",
       "      <td>Bash/Shell/PowerShell;C#;HTML/CSS;Java;Python;...</td>\n",
       "      <td>180000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88877</th>\n",
       "      <td>United States</td>\n",
       "      <td>Bash/Shell/PowerShell;C;Clojure;HTML/CSS;Java;...</td>\n",
       "      <td>2000000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88878</th>\n",
       "      <td>United States</td>\n",
       "      <td>HTML/CSS;JavaScript;Scala;TypeScript</td>\n",
       "      <td>130000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88879</th>\n",
       "      <td>Finland</td>\n",
       "      <td>Bash/Shell/PowerShell;C++;Python</td>\n",
       "      <td>82488.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88882</th>\n",
       "      <td>Netherlands</td>\n",
       "      <td>C#;HTML/CSS;Java;JavaScript;PHP;Python</td>\n",
       "      <td>588012.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>22289 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Country                                 LanguageWorkedWith  \\\n",
       "Respondent                                                                      \n",
       "6                   Canada                                         Java;R;SQL   \n",
       "9              New Zealand  Bash/Shell/PowerShell;C#;HTML/CSS;JavaScript;P...   \n",
       "13           United States  Bash/Shell/PowerShell;HTML/CSS;JavaScript;PHP;...   \n",
       "16          United Kingdom  Bash/Shell/PowerShell;C#;HTML/CSS;JavaScript;T...   \n",
       "22           United States  Bash/Shell/PowerShell;C++;HTML/CSS;JavaScript;...   \n",
       "...                    ...                                                ...   \n",
       "88876        United States  Bash/Shell/PowerShell;C#;HTML/CSS;Java;Python;...   \n",
       "88877        United States  Bash/Shell/PowerShell;C;Clojure;HTML/CSS;Java;...   \n",
       "88878        United States               HTML/CSS;JavaScript;Scala;TypeScript   \n",
       "88879              Finland                   Bash/Shell/PowerShell;C++;Python   \n",
       "88882          Netherlands             C#;HTML/CSS;Java;JavaScript;PHP;Python   \n",
       "\n",
       "            ConvertedComp  \n",
       "Respondent                 \n",
       "6                366420.0  \n",
       "9                 95179.0  \n",
       "13                90000.0  \n",
       "16               455352.0  \n",
       "22               103000.0  \n",
       "...                   ...  \n",
       "88876            180000.0  \n",
       "88877           2000000.0  \n",
       "88878            130000.0  \n",
       "88879             82488.0  \n",
       "88882            588012.0  \n",
       "\n",
       "[22289 rows x 3 columns]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[filt_highsal, ['Country', 'LanguageWorkedWith', 'ConvertedComp']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "countries = ['India', 'Germany']\n",
    "country_filt = df['Country'].isin(countries)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LanguageWorkedWith</th>\n",
       "      <th>ConvertedComp</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Respondent</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Bash/Shell/PowerShell;C;C++;HTML/CSS;Java;Java...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>C#;Go;JavaScript;Python;R;SQL</td>\n",
       "      <td>13293.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>C++</td>\n",
       "      <td>57060.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Assembly;Bash/Shell/PowerShell;C;C++;HTML/CSS;...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>Java;Kotlin;Python</td>\n",
       "      <td>20628.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80319</th>\n",
       "      <td>Bash/Shell/PowerShell;HTML/CSS;Java;JavaScript...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83786</th>\n",
       "      <td>HTML/CSS;JavaScript;Kotlin;TypeScript</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83862</th>\n",
       "      <td>Assembly;C;C++;HTML/CSS;Java;JavaScript;Object...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84299</th>\n",
       "      <td>HTML/CSS;Java;JavaScript;Python;Swift;TypeScript</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>86012</th>\n",
       "      <td>Bash/Shell/PowerShell;C++;HTML/CSS;JavaScript</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>14927 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           LanguageWorkedWith  ConvertedComp\n",
       "Respondent                                                                  \n",
       "8           Bash/Shell/PowerShell;C;C++;HTML/CSS;Java;Java...            NaN\n",
       "10                              C#;Go;JavaScript;Python;R;SQL        13293.0\n",
       "14                                                        C++        57060.0\n",
       "15          Assembly;Bash/Shell/PowerShell;C;C++;HTML/CSS;...            NaN\n",
       "36                                         Java;Kotlin;Python        20628.0\n",
       "...                                                       ...            ...\n",
       "80319       Bash/Shell/PowerShell;HTML/CSS;Java;JavaScript...            NaN\n",
       "83786                   HTML/CSS;JavaScript;Kotlin;TypeScript            NaN\n",
       "83862       Assembly;C;C++;HTML/CSS;Java;JavaScript;Object...            NaN\n",
       "84299        HTML/CSS;Java;JavaScript;Python;Swift;TypeScript            NaN\n",
       "86012           Bash/Shell/PowerShell;C++;HTML/CSS;JavaScript            NaN\n",
       "\n",
       "[14927 rows x 2 columns]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# df[country_filt, ['LanguageWorkedWith', 'ConvertedComp'] ] this gives error\n",
    "df.loc[country_filt, ['LanguageWorkedWith', 'ConvertedComp'] ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### String Methods Basics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "language filter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "lang_filt = df['LanguageWorkedWith'].str.contains('Python', na=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we can use `.str.contains()` and give `na=False` for those columns that contains no values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LanguageWorkedWith</th>\n",
       "      <th>ConvertedComp</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Respondent</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>HTML/CSS;Java;JavaScript;Python</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>C++;HTML/CSS;Python</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>C;C++;C#;Python;SQL</td>\n",
       "      <td>61000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>C++;HTML/CSS;Java;JavaScript;Python;SQL;VBA</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Bash/Shell/PowerShell;C;C++;HTML/CSS;Java;Java...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84539</th>\n",
       "      <td>Bash/Shell/PowerShell;C;C++;HTML/CSS;Java;Java...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>85738</th>\n",
       "      <td>Bash/Shell/PowerShell;C++;Python;Ruby;Other(s):</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>86566</th>\n",
       "      <td>Bash/Shell/PowerShell;HTML/CSS;Python;Other(s):</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87739</th>\n",
       "      <td>C;C++;HTML/CSS;JavaScript;PHP;Python;SQL</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88212</th>\n",
       "      <td>HTML/CSS;JavaScript;Python</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>36443 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           LanguageWorkedWith  ConvertedComp\n",
       "Respondent                                                                  \n",
       "1                             HTML/CSS;Java;JavaScript;Python            NaN\n",
       "2                                         C++;HTML/CSS;Python            NaN\n",
       "4                                         C;C++;C#;Python;SQL        61000.0\n",
       "5                 C++;HTML/CSS;Java;JavaScript;Python;SQL;VBA            NaN\n",
       "8           Bash/Shell/PowerShell;C;C++;HTML/CSS;Java;Java...            NaN\n",
       "...                                                       ...            ...\n",
       "84539       Bash/Shell/PowerShell;C;C++;HTML/CSS;Java;Java...            NaN\n",
       "85738         Bash/Shell/PowerShell;C++;Python;Ruby;Other(s):            NaN\n",
       "86566         Bash/Shell/PowerShell;HTML/CSS;Python;Other(s):            NaN\n",
       "87739                C;C++;HTML/CSS;JavaScript;PHP;Python;SQL            NaN\n",
       "88212                              HTML/CSS;JavaScript;Python            NaN\n",
       "\n",
       "[36443 rows x 2 columns]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[lang_filt, ['LanguageWorkedWith', 'ConvertedComp']]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}