notebooks/project_settings_module.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "julian-vertical",
   "metadata": {},
   "source": [
    "# Use a project settings module\n",
    "\n",
    "I always struggle to keep track of paths in my project scripts and notebooks. Are they relative to the notebook file? Or the directory I was in when I started the notebook server?\n",
    "\n",
    "To make this explicit, I started adding a settings module to my project similar to the [one used in Django projects](https://docs.djangoproject.com/en/3.2/topics/settings/). Django is a popular Python web application framework, so this convention is likely to be easily understood by other Python developers.\n",
    "\n",
    "You can see a bare-bones one in the [`settings.py`](../settings.py) file at the root of this project's repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "democratic-spanking",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"\"\"Project settings\"\"\"\n",
      "\n",
      "from pathlib import Path\n",
      "\n",
      "PROJECT_ROOT = Path(__file__).parent\n",
      "# I use a layout similar to the one used by templates distributed along with AP's DataKit\n",
      "DATA_DIR = PROJECT_ROOT / \"data\"\n",
      "DATA_DIR_SRC = DATA_DIR / \"source\"\n",
      "DATA_DIR_MANUAL = DATA_DIR / \"manual\"\n",
      "DATA_DIR_PROCESSED = DATA_DIR / \"processed\"\n",
      "DATA_DIR_PUBLIC = DATA_DIR / \"public\""
     ]
    }
   ],
   "source": [
    "# Take a look at the settings module contents\n",
    "!cat settings.py"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "floating-freight",
   "metadata": {},
   "source": [
    "However, before you can import this module in your notebook, you have to tell Python where to find the module. I like to do this using a script that is run when Python is started using the [`PYTHONSTARTUP` environment variable](http://www.witkowskibartosz.com/blog/pythonstartup_what_it_is_and_how_to_use_it.html).\n",
    "\n",
    "This script changes directory to the project root.\n",
    "\n",
    "If you're using this in your project, you want to change the `project_name` variable to reflect your project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "young-newman",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "import os\n",
      "\n",
      "project_name = \"python-data-cheatsheet\" \n",
      "cwd = os.getcwd()\n",
      "basename = os.path.basename(cwd)\n",
      "project_root_not_found = False\n",
      "\n",
      "while basename != project_name:\n",
      "    if os.path.split(cwd)[1] == '':\n",
      "        basename = project_name\n",
      "        project_root_not_found = True\n",
      "    else:\n",
      "        cwd = os.path.split(cwd)[0]\n",
      "        basename = os.path.basename(cwd)\n",
      "\n",
      "if project_root_not_found:\n",
      "    print(\"Couldn't find project root. Current directory not changed to project root.\")\n",
      "else:\n",
      "    os.chdir(cwd)\n"
     ]
    }
   ],
   "source": [
    "# Show the Python startup script\n",
    "!cat .startup.py"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "spanish-grave",
   "metadata": {},
   "source": [
    "Finally, you need to set the `PYTHONSTARTUP` environment variable and point it to your script.\n",
    "\n",
    "You'll want to replace `/home/ghing/workspace/python-data-cheatsheet/` with the full absolute path to your project directory.\n",
    "\n",
    "You can set the `PYTHONSTARTUP` environment variable in a number of ways.\n",
    "\n",
    "You could specify it when you start the notebook server:\n",
    "\n",
    "```\n",
    "PYTHONSTARTUP=/home/ghing/workspace/python-data-cheatsheet/.startup.py jupyter-lab\n",
    "```\n",
    "\n",
    "or export it and then it will be used any time you run Python in your current terminal\n",
    "\n",
    "```\n",
    "export PYTHONSTARTUP=/home/ghing/workspace/python-data-cheatsheet/.startup.py\n",
    "# Anything you run after this point will use that startup script\n",
    "```\n",
    "\n",
    "The way I do it, because I use [Pipenv](https://github.com/pypa/pipenv) to manage virtual environments for my project, and it automatically sources `.env` files when you run a command with `pipenv run`, is to just set the environment variable in a `.env` file in my project root."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "threaded-subcommittee",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PYTHONSTARTUP=/home/ghing/workspace/python-data-cheatsheet/.startup.py\n"
     ]
    }
   ],
   "source": [
    "# Show relevent line of .env file\n",
    "!grep PYTHONSTARTUP .env"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "considerable-coordination",
   "metadata": {},
   "source": [
    "This might seem like a lot to set up, but it's worth it to not have to think about mid-project. If you have one main project/virtual environment for your data analysis, you can do this once and set it and forget it.\n",
    "\n",
    "If you create many projects, I highly recommend the [DataKit](https://datakit.ap.org/) tool which will initialize all of this whenever you bootstrap a new project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "silver-perception",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setup\n",
    "\n",
    "import os\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# Import the settings module\n",
    "import settings"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "wrapped-personal",
   "metadata": {},
   "source": [
    "Then you can easily use the paths to data directories defined in your settings module to build paths to data that you open or save in your notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "exterior-visiting",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PosixPath('/home/ghing/workspace/python-data-cheatsheet/data/processed/test.csv')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Show building paths based on ones defined in the settings module\n",
    "\n",
    "settings.DATA_DIR_PROCESSED / \"test.csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "loose-station",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>A</th>\n",
       "      <th>B</th>\n",
       "      <th>C</th>\n",
       "      <th>D</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>NaN</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     A    B   C  D\n",
       "0  NaN  2.0 NaN  0\n",
       "1  3.0  4.0 NaN  1\n",
       "2  NaN  NaN NaN  5\n",
       "3  NaN  3.0 NaN  4"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create some toy data\n",
    "\n",
    "df = pd.DataFrame(\n",
    "    [\n",
    "        [np.nan, 2, np.nan, 0],\n",
    "        [3, 4, np.nan, 1],\n",
    "        [np.nan, np.nan, np.nan, 5],\n",
    "        [np.nan, 3, np.nan, 4],\n",
    "    ],\n",
    "    columns=list(\"ABCD\"),\n",
    ")\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "damaged-dance",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use a path in the settings module to unambiguously save the dataframe\n",
    "# to a known location.\n",
    "os.makedirs(settings.DATA_DIR_PROCESSED, exist_ok=True)\n",
    "df.to_csv(settings.DATA_DIR_PROCESSED / \"test.csv\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "environmental-living",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A,B,C,D\n",
      ",2.0,,0\n",
      "3.0,4.0,,1\n",
      ",,,5\n",
      ",3.0,,4\n"
     ]
    }
   ],
   "source": [
    "# This and remaining cells aren't likely to be things you would put in a real\n",
    "# notebook. They're just to show how this works.\n",
    "\n",
    "# Show that we successfully saved the file\n",
    "!cat data/processed/test.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "silver-vacuum",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cleanup\n",
    "!rm -f data/processed/test.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "instant-absence",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "python-data-cheatsheet",
   "language": "python",
   "name": "python-data-cheatsheet"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}