notebooks/3-advanced-usage-part-II.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Advanced Usage II\n",
    "\n",
    "**Concepts:**\n",
    "- Type Validation\n",
    "- `validator` decorator\n",
    "- `root_validator` decorator\n",
    "- Skipping validation\n",
    "- `validate_arguments` decorator\n",
    "\n",
    "\n",
    "## Type Validation\n",
    "\n",
    "We saw a sneak peak of type validation in Part 2. Basic Usage. Now we are going to take a deep dive in type validation with Pydantic.\n",
    "\n",
    "### `validator` decorator\n",
    "\n",
    "If custom validation is required above and beyond what Pydantic provides out of the box, the `validator` decorator may be used to create validation class methods as part of the Pydantic model defintion. For example, imagine we want to create a model representing a user, with fields for the user's given name, surname, username, and passwords from two separate password creation inputs. We may want to impose the following restrictions:\n",
    "1. Username must contain only ascii characters\n",
    "1. Both provided passwords must be identical\n",
    "1. Given name and surname must be alphabetic characters only and must start with a capital letter\n",
    "\n",
    "We can accomplish this using the following model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Any\n",
    "from pydantic import BaseModel, validator\n",
    "\n",
    "\n",
    "class User(BaseModel):\n",
    "    username: str\n",
    "    password1: str\n",
    "    password2: str\n",
    "    given_name: str\n",
    "    surname: str\n",
    "\n",
    "    @validator(\"username\")\n",
    "    def username_must_be_ascii(cls, username: str) -> str:\n",
    "        if not username.isascii():\n",
    "            raise ValueError(\"must be alphanumeric\")\n",
    "        return username\n",
    "\n",
    "    @validator(\"password2\")\n",
    "    def passwords_must_match(cls, password2: str, values: dict[str, Any]) -> str:\n",
    "        if (\"password1\" in values) and (password2 != values[\"password1\"]):\n",
    "            raise ValueError(\"Passwords do not match\")\n",
    "        return password2\n",
    "\n",
    "    @validator(\"given_name\", \"surname\")\n",
    "    def names_must_be_alphabetic(cls, name: str) -> str:\n",
    "        if not name.isalpha():\n",
    "            raise ValueError(\"must be alphabetic\")\n",
    "        return name.capitalize()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's create a valid user and see what happens..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display(\n",
    "    User(\n",
    "        username=\"scipy.2023.is.fun\",\n",
    "        password1=\"sup3rSecurePa$$w0rd\",\n",
    "        password2=\"sup3rSecurePa$$w0rd\",\n",
    "        given_name=\"joHn\",\n",
    "        surname=\"doe\",\n",
    "    )\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that even though the provided name \"joHn doe\" was not properly capitalized, we were able to correct this by using the `str.capitalize` method and thus did not need to throw any errors. Now let's create an invalid user and see what happens..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "raises-exception"
    ]
   },
   "outputs": [],
   "source": [
    "display(\n",
    "    User(\n",
    "        username=\"§cipy.2023.is.fun\",\n",
    "        password1=\"sup3rSecurePa$$w0rd\",\n",
    "        password2=\"sup3rSecurePa$$w0rd2\",\n",
    "        given_name=\"John Harry\",\n",
    "        surname=\"Doe-Smith\",\n",
    "    )\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that Pydantic does not stop at the first validation error. It keeps track of all validation errors and gives a detailed summary of all validation errors in one error message. This allows the user to fix multiple problems at once instead of fixing a problem and then running into the next problem.\n",
    "\n",
    "Some notes to keep in mind when using the `validator` decorator:\n",
    "* The name of the validation method can be any valid Python name, but it helps to be descriptive\n",
    "* The method will be a class method and not an instance method, so it is customary to name the first methd parameter `cls`\n",
    "* The second method parameter will refer to the parsed value of the field under inspection and can be any valid Python name\n",
    "* An optional third method parameter called `values` will refer to a dictionary of all previously parsed fields (fields are parsed in the order they are defined in the model)\n",
    "  * If a field fails validation, it will not be present in the `values` dictionary in remaining validation methods\n",
    "* The same method may be used to validate multiple fields by passing the name of each field as multiple arguments to the decorator\n",
    "* Validation methods can perform additional parsing of fields on top of any parsing automatically provided by Pydantic\n",
    "* Validation methods should either return the parsed value or raise one of `ValueError`, `TypeError`, or `AssertionError`\n",
    "\n",
    "By default, `validator` will perform validation *after* other validation such as coercing `\"5\"` to an `int`. But we can create validation methods that operate *before* other validation by using `pre=True` in the `validator` keyword arguments.\n",
    "\n",
    "`validator` also has an `each_item` keyword argument that will apply the method to each item of a list-like field.\n",
    "\n",
    "Let's see `pre` and `each_item` in action...\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "raises-exception"
    ]
   },
   "outputs": [],
   "source": [
    "class Foo(BaseModel):\n",
    "    positive_ints: list[int]\n",
    "\n",
    "    @validator(\"positive_ints\", pre=True)\n",
    "    def split_comma_separated_values(cls, positive_ints: Any) -> Any:\n",
    "        if isinstance(positive_ints, str):\n",
    "            return positive_ints.split(\",\")\n",
    "        return positive_ints\n",
    "\n",
    "    @validator(\"positive_ints\", each_item=True)\n",
    "    def must_be_positive(cls, item: int) -> int:\n",
    "        if item <= 0:\n",
    "            raise ValueError(f\"{item} is not positive\")\n",
    "        return item\n",
    "\n",
    "\n",
    "display(\n",
    "    Foo(positive_ints=(67.4, 2, True)),\n",
    ")\n",
    "display(\n",
    "    Foo(positive_ints=\"2,4,6,8\"),\n",
    ")\n",
    "display(\n",
    "    Foo(positive_ints=[\"-4\", 4, 0, 7]),\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the first example successfully converted `(67.4, 2, True)` into the list of integers `[67, 2, 1]`. And thanks to `pre=True`, the second example converted `\"2,4,6,8\"` to `[2, 4, 6, 8]`. Had we set `pre=False` or left the default value, the string `\"2,4,6,8\"` would have led to a validation error. The third example returns a validation error as the input list contains non-positive integers. The error message even tells us which indices of the input list are leading to the validation error.\n",
    "\n",
    "Another default behavior of `validator` is to not validate fields when a value is not provided. But there may be scenarios where this is not the desired behavior. Let's see an example. Let's return to our `User` class from above. Let's create a `NewUser` class which is the same as the `User` class but contains an additional `created_at` field. This will take an optional `datetime`. When not supplied, the default value should be the current UTC time. A naive implementation might look like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "\n",
    "class NewUser(User):\n",
    "    created_at: datetime = datetime.utcnow()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But there is a problem with this implementation. The default time will represent the time the class was defined, not when it was instantiated..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "\n",
    "class NewUser(User):\n",
    "    created_at: datetime = datetime.utcnow()\n",
    "\n",
    "\n",
    "print(f\"NewUser class defined at:     {datetime.utcnow()}\")\n",
    "\n",
    "time.sleep(3)\n",
    "\n",
    "new_user = NewUser(\n",
    "    username=\"joe\",\n",
    "    password1=\"1234\",\n",
    "    password2=\"1234\",\n",
    "    given_name=\"joe\",\n",
    "    surname=\"davis\",\n",
    ")\n",
    "print(f\"new_user instance created at: {new_user.created_at}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that despite occurring 3 seconds apart, the class definition and class instantiation are reporting as just milliseconds apart. We can solve this by creating a validation method where we set `always=True` in the `validator` keyword arguments. A correct implemention of this class looks like..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Optional, Union\n",
    "\n",
    "\n",
    "class NewUser(User):\n",
    "    created_at: Optional[datetime]\n",
    "\n",
    "    @validator(\"created_at\", always=True)\n",
    "    def set_default_time(cls, created_at: Union[datetime, None]) -> datetime:\n",
    "        return datetime.utcnow() if created_at is None else created_at\n",
    "\n",
    "\n",
    "print(f\"NewUser class defined at:     {datetime.utcnow()}\")\n",
    "\n",
    "time.sleep(3)\n",
    "\n",
    "new_user = NewUser(\n",
    "    username=\"joe\",\n",
    "    password1=\"1234\",\n",
    "    password2=\"1234\",\n",
    "    given_name=\"joe\",\n",
    "    surname=\"davis\",\n",
    ")\n",
    "print(f\"new_user instance created at: {new_user.created_at}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we get the expected behavior that the `new_user` was created 3 seconds after the `NewUser` class was defined.\n",
    "\n",
    "### `root_validator` decorator\n",
    "\n",
    "It is possible to perform validation on the entire model data in one validation method using the `root_validator` decorator. Recall in our `User` class we validated that `password1` and `password2` matched using the `validator` decorator. The same functionality can be implemented with the `root_validator` decorator..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "raises-exception"
    ]
   },
   "outputs": [],
   "source": [
    "from pydantic import root_validator\n",
    "\n",
    "\n",
    "class UserPassword(BaseModel):\n",
    "    password1: str\n",
    "    password2: str\n",
    "\n",
    "    @root_validator\n",
    "    def passords_must_match(cls, values: dict[str, Any]) -> dict[str, Any]:\n",
    "        pw1, pw2 = values.get(\"password1\"), values.get(\"password2\")\n",
    "        if pw1 is not None and pw2 is not None and pw1 != pw2:\n",
    "            raise ValueError(\"passwords do not match\")\n",
    "        return values\n",
    "\n",
    "\n",
    "display(\n",
    "    UserPassword(password1=\"1234\", password2=\"1234\"),\n",
    ")\n",
    "display(\n",
    "    UserPassword(password1=\"1234\", password2=\"12345\"),\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Root validators also accept a `pre=True` keyword argument just like the `validator` decorator.\n",
    "\n",
    "The full documentation on validators can be found at https://docs.pydantic.dev/latest/usage/validators/\n",
    "\n",
    "### Skipping validation\n",
    "\n",
    "Type validation can be a slow process that you may want to skip for performance reasons. If you know you have data from a trusted source that is pre-validated, then you may use the `construct` method of your Pydantic model when instantiating the class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_data = {\n",
    "    \"username\": \"scipy.2023.is.fun\",\n",
    "    \"password1\": \"1234\",\n",
    "    \"password2\": \"1234\",\n",
    "    \"given_name\": \"john\",\n",
    "    \"surname\": \"doe\",\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%timeit\n",
    "User(**user_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%timeit\n",
    "User.construct(**user_data)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that instatiating the `User` class with `construct` is much faster than with validation. But be mindful that skipping validation can result in invalid field values if the data is not pre-validated. Look what happens if we use the previous example that led to 4 validation errors..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display(\n",
    "    User.construct(\n",
    "        username=\"§cipy.2023.is.fun\",\n",
    "        password1=\"sup3rSecurePa$$w0rd\",\n",
    "        password2=\"sup3rSecurePa$$w0rd2\",\n",
    "        given_name=\"John Harry\",\n",
    "        surname=\"Doe-Smith\",\n",
    "    )\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected we see no validation error and the resulting `User` instance has several invalid values.\n",
    "\n",
    "### `Config` type validation settings\n",
    "\n",
    "Recall we can customize our Pydantic model by setting certain attributes in the `Config` class in our model. There are several settings related to type validation that can reduce the amount of validation methods needed.\n",
    "* `anystr_strip_whitespace` - strip leading and trailing whitespaces for all `str` and `bytes` (default: `False`)\n",
    "* `anystr_upper` - convert all characters to uppercase for `str` and `bytes` (default: `False`)\n",
    "* `anystr_lower` - convert all characters to lowercase for `str` and `bytes` (default: `False`)\n",
    "* `min_anystr_length` - minimum length for all `str` and `bytes` (default: `0`)\n",
    "* `max_anystr_length` - maximum length for all `str` and `bytes` (default: `None`)\n",
    "* `validate_all` - whether to validate fields when no value is provided (default: `False`)\n",
    "* `validate_assignment` - whether to validate when fields are updated after instantiation (default: `False`)\n",
    "\n",
    "Let's see some of these in action..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Foo(BaseModel):\n",
    "    bar: str\n",
    "\n",
    "    class Config:\n",
    "        anystr_strip_whitespace = True\n",
    "        anystr_upper = True\n",
    "        min_anystr_length = 8\n",
    "        max_anystr_length = 32\n",
    "        validate_assignment = True\n",
    "\n",
    "\n",
    "display(\n",
    "    Foo(bar=\"   hello SciPy!      \"),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "raises-exception"
    ]
   },
   "outputs": [],
   "source": [
    "display(\n",
    "    Foo(bar=\"    baz   \"),\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that in this example the provided string has 10 characters, surpassing the 8 character minimum, but after stripping leading and trailing whitespaces, the string only has 3 characters. Thus a validation error arises exclaiming that `bar` is not at least 8 characters long."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "raises-exception"
    ]
   },
   "outputs": [],
   "source": [
    "foo = Foo(bar=\"   hello SciPy!      \")\n",
    "foo.bar = 80 * \"-\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we try to change the value of `foo.bar` after instantiation, but we get a validation error exclaiming that the provided string is too long. Note that if `validate_assignment` were set to `False`, this would not have raised a validation error.\n",
    "\n",
    "### `validate_arguments` decorator\n",
    "\n",
    "Until now we have been discussing type validation in the context of Pydantic models, but Pydantic also provides a way to enforce function argument types at runtime via the `validate_arguments` decorator. This decorator will perform type coercion and type validation just like any Pydantic model.\n",
    "\n",
    "Continuing our example with the `User` class, suppose a user wants to login to a service and we want to validate the provided credentials. We could write a function `validate_credentials` to compare a user provided username/password pair to a list of `User` instances. We may want to be sure the provided type are of the expected types.\n",
    "\n",
    "**DISCLAIMER:** The following is **not** intended to be used as a model for sensitive date storage and/or credentials authentication. This example serves only to demonstrate the usage of the `validate_arguments` decorator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pydantic import validate_arguments\n",
    "\n",
    "\n",
    "def login():\n",
    "    print(\"Successfully logged in!\")\n",
    "\n",
    "\n",
    "def logout():\n",
    "    print(\"Invalid username or password\")\n",
    "\n",
    "\n",
    "@validate_arguments\n",
    "def validate_credentials(\n",
    "    username: str,\n",
    "    password: str,\n",
    "    registered_users: list[User],\n",
    ") -> None:\n",
    "    for user in registered_users:\n",
    "        if (user.username == username) and (user.password1 == password):\n",
    "            login()\n",
    "            return\n",
    "\n",
    "    logout()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's create a database of users, but we will not create any `User` classes. Each user will just be a dictionary that conforms to the `User` model definition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "registered_users = [\n",
    "    {\n",
    "        \"username\": \"abc\",\n",
    "        \"password1\": \"123\",\n",
    "        \"password2\": \"123\",\n",
    "        \"given_name\": \"Joe\",\n",
    "        \"surname\": \"Smith\",\n",
    "    },\n",
    "    {\n",
    "        \"username\": \"def\",\n",
    "        \"password1\": \"321\",\n",
    "        \"password2\": \"321\",\n",
    "        \"given_name\": \"Jane\",\n",
    "        \"surname\": \"Davidson\",\n",
    "    },\n",
    "]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's attempt to login in to the service..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "username, password = \"def\", \"321\"\n",
    "\n",
    "validate_credentials(\n",
    "    username=username,\n",
    "    password=password,\n",
    "    registered_users=registered_users,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We successfully logged in! But notice the magic that happened here. Our database of users is just a list of dictionaries. None of the users in our database have `username` or `password1` attributes, yet our `validate_credentials` function was able to access these attributes without raising an exception. Evidently the `validate_arguments` decorator coerced the list of dictionaries into a list of `User` instances. Presumably, if we then alter our database to include invalid users, then we should get a validation error when we try to call `validate_credentials`. Let's see if we are right..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "raises-exception"
    ]
   },
   "outputs": [],
   "source": [
    "registered_users.append(\n",
    "    {\n",
    "        \"username\": \"ghi\",\n",
    "        \"password1\": \"pass\",\n",
    "        \"password2\": \"different-pass\",\n",
    "        \"given_name\": \"John\",\n",
    "        \"surname\": \"Johnson\",\n",
    "    },\n",
    ")\n",
    "\n",
    "validate_credentials(\n",
    "    username=username,\n",
    "    password=password,\n",
    "    registered_users=registered_users,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So even though we provided the same, valid username and password, the function did not complete and instead raised a validation error because it could not coerce the list of registered user dictionaries into a list of `User` instances.\n",
    "\n",
    "The `validate_arguments` decorator can provide peace of mind when developing code, because you know the input arguments will have the specified type inside the function body. As with any validation, this does take time so the drawback is a decrease in performance. Though, in practice, this drop in speed is almost never important.\n",
    "\n",
    "## Exercise 4\n",
    "\n",
    "Now let's extend the script from the previous exercise with validation functionality, to make sure we work with valid data. \n",
    "Here are some things to consider:\n",
    "\n",
    "- Are the time and temperature lists the same length?\n",
    "- Are all temperatures above absolute zero?\n",
    "- Are longitude and latitude values within the expected range?\n",
    "- Any other meaningful validation you can think of\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "scipy-2023-pydantic-tutorial",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}