notebooks/2-basic-usage.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "177a1666",
   "metadata": {},
   "source": [
    "# 2. Basic Usage\n",
    "\n",
    "**Concepts:**\n",
    "* Creation of a simple Pydantic model\n",
    "* Initializing models and setting model attributes\n",
    "* Type parsing\n",
    "* Atomic types\n",
    "* Type validation (sneak peak)\n",
    "* Default values and optional values\n",
    "* Model Config and Config.extra = “forbid”\n",
    "\n",
    "For comparison we fisrt start with the simple class definitons we saw in the introduction:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec44701d",
   "metadata": {},
   "outputs": [],
   "source": [
    "class PointV1:\n",
    "    \"\"\"Representation of a two-dimensional point coordinate.\"\"\"\n",
    "\n",
    "    def __init__(self, x, y):\n",
    "        \"\"\"Initializes a PointV1 with the given coordinates.\"\"\"\n",
    "        self.x = x\n",
    "        self.y = y\n",
    "\n",
    "    def distance_to(self, other):\n",
    "        \"\"\"Computes the distance to another `PointV1`.\"\"\"\n",
    "        dx = self.x - other.x\n",
    "        dy = self.y - other.y\n",
    "        return (dx**2 + dy**2) ** 0.5"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "283c10d3",
   "metadata": {},
   "source": [
    "And the version with type annotations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92b56a31",
   "metadata": {},
   "outputs": [],
   "source": [
    "class PointV2:\n",
    "    \"\"\"Representation of a two-dimensional point coordinate.\"\"\"\n",
    "\n",
    "    def __init__(self, x: float, y: float) -> None:\n",
    "        \"\"\"Initializes a PointV2 with the given coordinates.\"\"\"\n",
    "        self.x = x\n",
    "        self.y = y\n",
    "\n",
    "    def distance_to(self, other: \"PointV2\") -> float:\n",
    "        \"\"\"Computes the distance to another `PointV2`.\"\"\"\n",
    "        dx = self.x - other.x\n",
    "        dy = self.y - other.y\n",
    "        return (dx**2 + dy**2) ** 0.5"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ffcf13c4",
   "metadata": {},
   "source": [
    "\n",
    "## Pydantic models\n",
    "\n",
    "Now that we've reviewed type hints and class defintions, let's convert the above example to a Pydantic model. This will involve only a few changes and will noticeably improve the code readability even further..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8febbc2a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pydantic import BaseModel\n",
    "\n",
    "\n",
    "class PointV3(BaseModel):\n",
    "    \"\"\"Representation of a two-dimensional point coordinate.\"\"\"\n",
    "\n",
    "    x: float\n",
    "    y: float\n",
    "\n",
    "    def distance_to(self, other: \"PointV3\") -> float:\n",
    "        \"\"\"Computes the distance to another `PointV3`.\"\"\"\n",
    "        dx = self.x - other.x\n",
    "        dy = self.y - other.y\n",
    "        return (dx**2 + dy**2) ** 0.5\n",
    "\n",
    "\n",
    "p2 = PointV3(x=-1, y=4)\n",
    "display(p2)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f083a9d3",
   "metadata": {},
   "source": [
    "That's it! We only need to import `BaseModel` from Pydantic, use it as the parent class of `PointV3`, and list the instance attributes (with their type hints) as if they were class attributes. Pydantic automatically creates an `__init__` method for us using the provided class attributes and their type hints. Notice, Pydantic also provides a `__repr__` method that prints out a nice representation of `p2`.\n",
    "\n",
    "After instantiation of a Pydantic model, the instance will, under most circumstances, behave as a normal Python class. For example we can update attributes after instantiation, just as with a normal Python class..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71a67ba2",
   "metadata": {},
   "outputs": [],
   "source": [
    "p2 = PointV3(x=-1, y=4)\n",
    "display(p2)\n",
    "\n",
    "p2.y = 0\n",
    "display(p2)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "581cb794",
   "metadata": {},
   "source": [
    "## Type parsing\n",
    "\n",
    "Recall the bug we had earlier when trying to instantiate a `PointV1` with stringified integers. With Pydantic, this is no longer a bug! If the input arguments `x` and `y` are not of the expected type, Pydantic uses the argument type hints to attempt to coerce the input arguments into the expected types. Let's see this in action"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65038521",
   "metadata": {},
   "outputs": [],
   "source": [
    "x1, y1, x2, y2 = \"5\", \"7\", -1, 4\n",
    "\n",
    "p1_v1 = PointV1(x=x1, y=y1)\n",
    "p2_v1 = PointV1(x=x2, y=y2)\n",
    "\n",
    "p1_v3 = PointV3(x=x1, y=y1)\n",
    "p2_v3 = PointV3(x=x2, y=y2)\n",
    "\n",
    "for point in [p1_v1, p2_v1, p1_v3, p2_v3]:\n",
    "    print(\"\\n\", repr(point))\n",
    "    print(f\"{point.x = } and {type(point.x) = }\")\n",
    "    print(f\"{point.y = } and {type(point.y) = }\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "b2beef62",
   "metadata": {},
   "source": [
    "Notice that for `PointV1` (which is not a Pydantic model), the `x` and `y` attributes are exactly as provided, i.e. strings and integers. But for `PointV3` (which is a Pydantic model), the `x` and `y` attributes were converted to floats because they were type hinted as such. Pydantic applied the `float` constructor to the provided values i.e. `float(\"5\") == 5.0`, `float(\"7\") == 7.0`, `float(-1) == -1.0`, and `float(4) == 4.0`. Later, we will discuss the case when this conversion is not possible e.g. `float(\"hello world\")`.\n",
    "\n",
    "Note: `PointV1` and `PointV2` only differ in type hints and have no run-time difference. We would have obtained the same results with `PointV2` as we did with `PointV1`. Try it!\n",
    "\n",
    "We can further elucidate this point by attempting to compute the distances between the points for each version:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f061ab68",
   "metadata": {
    "tags": [
     "raises-exception"
    ]
   },
   "outputs": [],
   "source": [
    "p1_v1.distance_to(p2_v1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ea1376b5",
   "metadata": {},
   "source": [
    "Without Pydantic we get the expected error, but with Pydantic..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b562960c",
   "metadata": {},
   "outputs": [],
   "source": [
    "p1_v3.distance_to(p2_v3)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "808fcd59",
   "metadata": {},
   "source": [
    "we obtain the distance between the two points! This is thanks to Pydantic's type parsing.\n",
    "\n",
    "## Atomic types\n",
    "\n",
    "From this point on, I will refer to Pydantic model attributes as fields, as is done in the Pydantic documentation.\n",
    "\n",
    "Pydantic supports a large variety of field types but the most basic are `None`, `bool`, `int`, `float`, `str` and `bytes`.\n",
    "\n",
    "### `None`\n",
    "\n",
    "The only allowed value is `None`.\n",
    "\n",
    "### `bool`\n",
    "\n",
    "The allowed values for `True` are `{True, 1, \"1\", \"on\", \"t\", \"true\", \"y\", \"yes\"}`. The allowed values for `False` are `{False, 0, \"0\", \"off\", \"f\", \"false\", \"n\", \"no\"}`.\n",
    "\n",
    "### `int`\n",
    "\n",
    "`int(v)` will be used to coerce values to the `int` type. WARNING: This can lead to a loss of information. If `x` is type hinted as `int` and the provided value is `2.72`, the resulting value of `x` will be `2` because `int(2.72) == 2`.\n",
    "\n",
    "### `float`\n",
    "\n",
    "`float(v)` will be used to coerce values to the `float` type.\n",
    "\n",
    "### `str`\n",
    "\n",
    "`str(v)` will be used to coerce numeric types and `v.decode()` will be used to coerce `bytes` and `bytearray`.\n",
    "\n",
    "### `bytes`\n",
    "\n",
    "`bytes(v)` will be used to coerce `bytearray`, `v.encode()` will be used to coerce `str`, and `str(v).encode()` will be used to coerce numeric types.\n",
    "\n",
    "Let's see an example of type coercion for each of these types (with the exception of `None` which does not use coercion)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "323e13bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "class TypeCoercionExample(BaseModel):\n",
    "    v_bool: bool\n",
    "    v_int: int\n",
    "    v_float: float\n",
    "    v_str: str\n",
    "    v_bytes: bytes\n",
    "\n",
    "\n",
    "display(\n",
    "    TypeCoercionExample(\n",
    "        v_bool=\"no\",\n",
    "        v_int=25.7,\n",
    "        v_float=True,\n",
    "        v_str=False,\n",
    "        v_bytes=\"hello\",\n",
    "    )\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "b8cb5b06",
   "metadata": {},
   "source": [
    "As expected\n",
    "* `v_bool=\"no\"` was coerced to `v_bool=False`\n",
    "* `v_int=25.7` was coerced to `v_int=25`    (Note the loss of information mentioned above)\n",
    "* `v_float=True` was coerced to `v_float=1.0`\n",
    "* `v_str=False` was coerced to `v_str='False'`\n",
    "* `v_bytes=\"hello\"` was coerced to `v_bytes=b'hello'`\n",
    "\n",
    "A detailed description of supported types can be found in the \"Field Types\" documentation page: https://docs.pydantic.dev/latest/usage/types/\n",
    "\n",
    "## Type validation (sneak peak)\n",
    "\n",
    "What happens when type coercion fails? This brings us to one of the main benefits of using Pydantic -- type validation. We will be discussing this in much more detail in part 3 of this tutorial, but let's get a sense of what will happen now while type coercion is still fresh in our minds.\n",
    "\n",
    "Recall our earlier example of the two-dimensional coordinate points. If we provided invalid arguments for `PointV1` and `PointV2` (recall these were not Pydantic models), the classes would happily instantiate and we would not see an error until later when trying to compute the distance between two points. And the error message didn't really point to the root cause of the problem, which was the invalid arguments during instantiation. We also showed that when using a Pydantic model (`PointV3`), Pydantic would attempt to correct our mistake for us using type coercion. But what if we provide arguments where type coercion fails? Let's find out. Returning to the earlier example, let's now provide arguments that will not pass type coercion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9d4c099",
   "metadata": {
    "tags": [
     "raises-exception"
    ]
   },
   "outputs": [],
   "source": [
    "class PointV3(BaseModel):\n",
    "    \"\"\"Representation of a two-dimensional point coordinate.\"\"\"\n",
    "\n",
    "    x: float\n",
    "    y: float\n",
    "\n",
    "    def distance_to(self, other: \"PointV3\") -> float:\n",
    "        \"\"\"Computes the distance to another `PointV3`.\"\"\"\n",
    "        dx = self.x - other.x\n",
    "        dy = self.y - other.y\n",
    "        return (dx**2 + dy**2) ** 0.5\n",
    "\n",
    "\n",
    "PointV3(\n",
    "    x=5,\n",
    "    y=\"this string cannot be coerced to a float\",\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6f5f11dd",
   "metadata": {},
   "source": [
    "Pydantic throws an error at the root cause and provides a helpful message! The message tells us that the provided value for field `y` is not valid for type `float`. This is potentially a big time saver when it comes to debugging. As mentioned above we will be diving much deeper into type validation later, but hopefully you now have a sense of what it is and why it is useful.\n",
    "\n",
    "## Default and optional values\n",
    "\n",
    "We can provide default field values by providing an `=` followed by the default value. We can create optional fields by importing `typing.Optional` and surrounding the type hint with `Optional`. The value of optional fields when not supplied by the user will be `None`.\n",
    "\n",
    "Let's provide an example by creating a new class that represents a line segment consisting of two x-y coordinates and an optional label. The first x-y coordinate will have default values set to the origin."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3cd4442",
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Optional\n",
    "\n",
    "\n",
    "class LineSegment(BaseModel):\n",
    "    x1: float = 0.0\n",
    "    y1: float = 0.0\n",
    "    x2: float\n",
    "    y2: float\n",
    "    label: Optional[str]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e94bcc1e",
   "metadata": {},
   "source": [
    "So `x1` and `y1` will default to `0.0` if not provided, `x2` and `y2` are required, and `label` is a string that defaults to `None` if not provided. Let's create a few `LineSegment` examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc6551e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(\n",
    "    LineSegment(x1=5, y1=6, x2=1, y2=9, label=\"red\"),\n",
    ")\n",
    "display(\n",
    "    LineSegment(x2=1, y2=9, label=\"red\"),\n",
    ")\n",
    "display(\n",
    "    LineSegment(x2=1, y2=9),\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "37d80ba4",
   "metadata": {},
   "source": [
    "And we get the expected behavior.\n",
    "\n",
    "## Model configuration\n",
    "\n",
    "Pydantic provides custamizable options for models. The model just needs to define a class called `Config` as an attribute. The attributes of `Config` serve as a configuration for the model.\n",
    "\n",
    "As an example, let's explore what happens when you instantiate a Pydantic model with fields that are not part of the model definition. By default, Pydantic models will ignore these additional fields. Let's provide our `LineSegment` model with an unexpected field..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a0ff423",
   "metadata": {
    "tags": [
     "raises-exception"
    ]
   },
   "outputs": [],
   "source": [
    "display(\n",
    "    LineSegment(x2=1, y2=9, hello=\"world\"),\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "436944cb",
   "metadata": {},
   "source": [
    "As expected the field `hello` was ignored and not included in the instantiated model. But we can configure the model to either allow or forbid extra fields not defined in the class..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da0fe9e0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from pydantic import Extra\n",
    "\n",
    "\n",
    "class LineSegment(BaseModel):\n",
    "    x1: float = 0.0\n",
    "    y1: float = 0.0\n",
    "    x2: float\n",
    "    y2: float\n",
    "    label: Optional[str]\n",
    "\n",
    "    class Config:\n",
    "        extra = Extra.allow\n",
    "\n",
    "\n",
    "display(\n",
    "    LineSegment(x2=1, y2=9, hello=\"world\"),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6cc7c85",
   "metadata": {
    "tags": [
     "raises-exception"
    ]
   },
   "outputs": [],
   "source": [
    "class LineSegment(BaseModel):\n",
    "    x1: float = 0.0\n",
    "    y1: float = 0.0\n",
    "    x2: float\n",
    "    y2: float\n",
    "    label: Optional[str]\n",
    "\n",
    "    class Config:\n",
    "        extra = Extra.forbid\n",
    "\n",
    "\n",
    "display(\n",
    "    LineSegment(x2=1, y2=9, hello=\"world\"),\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "202e8503",
   "metadata": {},
   "source": [
    "Class inheritance works as expected with Pydantic models..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1891c2ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "class A(BaseModel):\n",
    "    x: int\n",
    "\n",
    "    class Config:\n",
    "        extra = Extra.forbid\n",
    "\n",
    "\n",
    "class B(A):\n",
    "    y: bool\n",
    "\n",
    "\n",
    "class C(BaseModel):\n",
    "    z: str\n",
    "\n",
    "\n",
    "print(f\"{A.Config.extra = }\\t{A.__fields__.keys() = }\")\n",
    "print(f\"{B.Config.extra = }\\t{B.__fields__.keys() = }\")\n",
    "print(f\"{C.Config.extra = }\\t{C.__fields__.keys() = }\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "eaa8081a",
   "metadata": {},
   "source": [
    "Notice that `B`, which inherits from `A`, inherits the field, `x`, and configuration, `extra = Extra.forbid`, of `A`, while `C`, which is a base Pydantic model, does not contain the field `x` and obtains the default value of `Extra.ignore` for the `extra` configuration.\n",
    "\n",
    "A detailed description of possible configurations can be found in the \"Model Config\" documentation page: https://docs.pydantic.dev/latest/usage/model_config/\n",
    "\n",
    "## Exercise 2\n",
    "\n",
    "Consider the following set of weather data for the city of Murmansk, Russia:\n",
    "\n",
    "```python\n",
    "data_samples = [\n",
    "    {\n",
    "        \"date\": \"2023-05-20\",\n",
    "        \"temperature\": 62.2,\n",
    "        \"isCelsius\": False,\n",
    "        \"airQualityIndex\": \"24\",\n",
    "        \"sunriseTime\": \"01:26\",\n",
    "        \"sunsetTime\": \"00:00\",\n",
    "    },\n",
    "    {\n",
    "        \"date\": \"2023-05-21\",\n",
    "        \"temperature\": \"64.4\",\n",
    "        \"isCelsius\": \"not true\",\n",
    "        \"airQualityIndex\": 23,\n",
    "        \"sunriseTime\": \"01:10\",\n",
    "        \"sunsetTime\": \"00:16\",\n",
    "    },\n",
    "    {\n",
    "        \"date\": \"2023-05-22\",\n",
    "        \"temperature\": 14.4,\n",
    "        \"airQualityIndex\": 21,\n",
    "    },\n",
    "]\n",
    "```\n",
    "\n",
    "Write a script that computes and prints the average temperature (in celsius) in Murmansk for the dates provided. In your script, you should include a Pydantic model capable of parsing a single sample from the list of data samples above. You should parse each data sample one at a time using a `for` loop. (We will see how to parse the entire list simultaneously in the next section of this tutorial.)\n",
    "\n",
    "Some potentially helpful notes:\n",
    "* Some samples are missing data. You should decide how to handle these without inserting new data yourself (but modification of existing data is OK). Hint: refer to the \"Default and optional values\" section \n",
    "* Some values may produce errors and may need modification.\n",
    "* You may parse dates and times as strings for this exercise. (We will see a better way to parse dates and times later in this tutorial.)\n",
    "* You may assume `airQualityIndex` is an integer valued categorical variable."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "scipy-2023-pydantic-tutorial",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}