Draft: Reusing schema validators and serializers

[sydney-runkle]

Initial perf report, just with vals:

Need to work on error handling, serializers, and debugging.

Summary:

• The number of memory allocation has reduced 3.3x
• 35% less memory allocated

(pydantic) sydney-runkle@Sydneys-Pydantic-MBP pydantic % memray stats k8s_v2.bin
📏 Total allocations:
        1969609

📦 Total memory allocated:
        786.890MB

📊 Histogram of allocation size:
        min: 1.000B
        ---------------------------------------------
        < 4.000B   : 340813 ▇▇▇▇▇▇▇▇▇▇▇▇
        < 21.000B  : 747070 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇
        < 100.000B : 339933 ▇▇▇▇▇▇▇▇▇▇▇▇
        < 466.000B : 120403 ▇▇▇▇▇
        < 2.114KB  : 363496 ▇▇▇▇▇▇▇▇▇▇▇▇▇
        < 9.824KB  :  56606 ▇▇
        < 45.646KB :    991 ▇
        < 212.084KB:    124 ▇
        < 985.395KB:     50 ▇
        <=4.471MB  :    123 ▇
        ---------------------------------------------
        max: 4.471MB

📂 Allocator type distribution:
         MALLOC: 1826508
         REALLOC: 100096
         CALLOC: 42912
         MMAP: 93

🥇 Top 5 largest allocating locations (by size):
        - _get_code_from_file:<frozen runpy>:259 -> 282.632MB
        - create_schema_validator:/Users/sydney-runkle/Work/oss/pydantic/pydantic/plugin/_schema_validator.py:51 -> 187.160MB
        - complete_model_class:/Users/sydney-runkle/Work/oss/pydantic/pydantic/_internal/_model_construction.py:611 -> 133.740MB
        - __init__:/Users/sydney-runkle/.local/share/uv/python/cpython-3.13.0-macos-aarch64-none/lib/python3.13/typing.py:1035 -> 21.739MB
        - from_field:/Users/sydney-runkle/Work/oss/pydantic/pydantic/fields.py:279 -> 13.876MB

🥇 Top 5 largest allocating locations (by number of allocations):
        - create_schema_validator:/Users/sydney-runkle/Work/oss/pydantic/pydantic/plugin/_schema_validator.py:51 -> 1241381
        - complete_model_class:/Users/sydney-runkle/Work/oss/pydantic/pydantic/_internal/_model_construction.py:611 -> 507059
        - _get_code_from_file:<frozen runpy>:259 -> 74830
        - _apply_annotations:/Users/sydney-runkle/Work/oss/pydantic/pydantic/_internal/_generate_schema.py:2098 -> 30023
        - from_field:/Users/sydney-runkle/Work/oss/pydantic/pydantic/fields.py:279 -> 18946
(pydantic) sydney-runkle@Sydneys-Pydantic-MBP pydantic % memray stats k8s_v2-new.bin                                               
📏 Total allocations:
        591314

📦 Total memory allocated:
        518.786MB

📊 Histogram of allocation size:
        min: 1.000B
        ---------------------------------------------
        < 4.000B   :  90764 ▇▇▇▇▇▇▇▇▇▇▇▇▇
        < 21.000B  : 150994 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇
        < 100.000B : 103455 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇
        < 466.000B :  28960 ▇▇▇▇▇
        < 2.114KB  : 178561 ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇
        < 9.824KB  :  37417 ▇▇▇▇▇▇
        < 45.646KB :    911 ▇
        < 212.084KB:     96 ▇
        < 985.395KB:     42 ▇
        <=4.471MB  :    114 ▇
        ---------------------------------------------
        max: 4.471MB

📂 Allocator type distribution:
         MALLOC: 515841
         CALLOC: 41977
         REALLOC: 33412
         MMAP: 84

🥇 Top 5 largest allocating locations (by size):
        - _get_code_from_file:<frozen runpy>:259 -> 282.632MB
        - create_schema_validator:/Users/sydney-runkle/Work/oss/pydantic/pydantic/plugin/_schema_validator.py:51 -> 38.422MB
        - complete_model_class:/Users/sydney-runkle/Work/oss/pydantic/pydantic/_internal/_model_construction.py:611 -> 25.064MB
        - __init__:/Users/sydney-runkle/.local/share/uv/python/cpython-3.13.0-macos-aarch64-none/lib/python3.13/typing.py:1035 -> 21.739MB
        - from_field:/Users/sydney-runkle/Work/oss/pydantic/pydantic/fields.py:279 -> 13.876MB

🥇 Top 5 largest allocating locations (by number of allocations):
        - create_schema_validator:/Users/sydney-runkle/Work/oss/pydantic/pydantic/plugin/_schema_validator.py:51 -> 233203
        - complete_model_class:/Users/sydney-runkle/Work/oss/pydantic/pydantic/_internal/_model_construction.py:611 -> 109213
        - _get_code_from_file:<frozen runpy>:259 -> 74830
        - _apply_annotations:/Users/sydney-runkle/Work/oss/pydantic/pydantic/_internal/_generate_schema.py:2098 -> 30024
        - _extract_json_schema_info_from_field_info:/Users/sydney-runkle/Work/oss/pydantic/pydantic/_internal/_generate_schema.py:258 -> 30022

[sydney-runkle]

This logic is currently very wrong, I'm doing the arc stuff in the wrong place.

[sydney-runkle]

Basically, I need to move this up one level. I have hopes that this will further reduce memory usage. The stats above can be ignored for now...

[sydney-runkle]

Moving work over to #1616. As mentioned above, we need to move this one level higher.