ell serialization refactor / distributed 2 #362

alex-dixon · 2024-11-02T17:49:06Z

Rough overview:

Production environments may benefit from an architecture where lmp versions and invocation traces are written from multiple machines that do not share the same file system
Ideally, these setups need not miss out on any of ell’s features. This includes realtime updates to ell studio
Data stores like postgres do not offer an easy way to subscribe to realtime changes. We could have an api that works off postgres change data capture. I’ve worked with this extensively to implement materialized views from changes to tables using debezium, Kafka/redpanda, and Apache Spark. It is doable, but at the end of it this would at most support realtime for one datastore. Ideally ell and its feature set should be available with other data stores, even nosql ones. Allowing studio to wire itself to an event bus lets us “read our own writes” without getting into the particulars of realtime options per data store. This makes it much easier to integrate data stores because the realtime component is solved for all of them by default.
The event bus itself is pluggable in the same way as SQLite or postgres is for storage backends.
We establish patterns for adding new functionality via conditional import of modules and Ell’s —extras. We remove barriers to adoption and promote what is more or less single file contributions to support additional event buses or data stores. See the celery library for an example of this pattern scaling to dozens of different backends: https://github.com/celery/celery/tree/main/celery/backends
We continue to separate of ell studio from the core and lay groundwork for async support by introducing a serialization interface that is fully async and does not use sqlmodels as input or necessarily depend on sqlalchemy. Of all the changes this is the only one that touches the core and is probably the hardest to get right. The interfaces are written and ready for use but I have held off integrating them until we gain consensus on the code that is written here.
Even with serialization code added to core, ell users should notice 0 changes when updating the library except perhaps the addition of new configuration options. All changes are internal.

alex-dixon · 2024-11-17T14:36:28Z

src/ell/api/pubsub/abc.py

+Subscriber = WebSocket
+
+
+class PubSub(ABC):


Needs documented

alex-dixon · 2024-11-17T14:38:58Z

src/ell/api/pubsub/topic.py

+
+MAX_TOPIC_LENGTH = 65535
+
+class TopicMatcher:


alex-dixon · 2024-11-17T14:41:02Z

src/ell/api/server.py

+    app = FastAPI(
+        title="ell api",
+        description="ell api server",
+        version="0.1.0",


Should be ell version

alex-dixon · 2024-11-19T14:46:55Z

examples/future/images_minio.py

+
+
+# Load the image using PIL
+big_picture = Image.open(os.path.join(os.path.dirname(__file__), "bigpicture.jpg"))


Maybe download the big picture instead of keeping in repo

alex-dixon · 2024-11-19T15:10:33Z

src/ell/lmp/tool.py

@@ -97,7 +97,7 @@ def wrapper(

            # Determine the type annotation
            if param.annotation == inspect.Parameter.empty:
-                raise ValueError(f"Parameter {param_name} has no type annotation, and cannot be converted into a tool schema for OpenAI and other provisders. Should OpenAI produce a string or an integer, etc, for this parameter?")
+                raise ValueError(f"Parameter {param_name} has no type annotation, and cannot be converted into a tool schema for OpenAI and other providers. Should OpenAI produce a string or an integer, etc, for this parameter?")


Something like

Suggested change

raise ValueError(f"Parameter {param_name} has no type annotation, and cannot be converted into a tool schema for OpenAI and other providers. Should OpenAI produce a string or an integer, etc, for this parameter?")

raise ValueError(f"The {tool_name} tool parameter {param_name} needs a type annotation.

Example:

def {tool_name}({param_name}: str):

^^^ Add the type of {param_name} here

Tool function annotations help ell create a tool schema for OpenAI and other providers.")

Last line of the error message should be: More info: https://docs.ell.so/core_concepts/tool_usage.html#defining-tools

alex-dixon · 2024-11-19T15:21:24Z

src/ell/providers/bedrock.py

@@ -1,6 +1,9 @@
 from abc import ABC, abstractmethod
 from collections import defaultdict
 from typing import Any, Callable, Dict, List, Optional, Tuple, Type, Union, cast
+


Can be removed

src/ell/serialize/http.py

…ment

alex-dixon · 2024-11-20T16:17:11Z

src/ell/serialize/http.py

+#     """Serializes ell objects to json for writing to the database or wire protocols"""
+#     return json.dumps(
+#         pydantic_ltype_aware_cattr.unstructure(obj),
+#         sort_keys=True, default=repr, ensure_ascii=False)


Address this…will had a comment about serialize_object elsewhere

alex-dixon · 2024-11-20T16:20:05Z

src/ell/stores/models/core.py

 from typing import  Any
+import ell.types.serialize
+
+def utc_now() -> datetime:


See if this lives elsewhere and maybe name current_timestamp or something less awkward

alex-dixon added 30 commits November 2, 2024 10:04

add docker files

0c12db6

add serialization types

2a36678

add get_lmp method to store interface

1880d66

implement get_lmp, support in-memory sqlite for testing

8d5e72f

create sqlmodels from serialized

d70adbd

add serialization clients

1d840fd

add api server, tests

bc77671

start add api extras

29dd707

add dockerignore

59bf4cb

fix typo

bea7d6a

add docker build arguments with ell extras

8e8428d

add missing api-server main

5298d41

sidestep package not installed error

ed43769

fix: import base model from pydantic

c2fa5c7

checkpoint

d73f17b

default required deps in studio, api server images

20533ba

allow api.pubsub module so code can be shared

918cd4a

add api.client module

0e26714

fix imports

523edb2

centralize missing extras error

cd0fcf8

add serialize module, minio blob store

7f82ee8

remove unnecessary cast

3f39f86

update tests

e39cb6d

rename client to serializer

b2d5d84

add serializer to core

a71c28f

port invocation contents serialize fixes

651fbfd

json serialization over http

1209cd8

Merge branch 'main' into distributed

0c6e738

fix typo

ff3327b

tool call params dict or basemodel

3dc4344

alex-dixon added 8 commits November 17, 2024 11:38

properly serialize tool call params in providers

26d1eeb

simplify params serialization, add tool reference

1276eb6

content block parsed serde

7c935db

Merge branch 'main' into distributed

4745314

refactor(serialize): remove redundant code in sql store wrappers

e572492

fix: created_at not optional if defaulted

ff609fd

naming: use 'coerce' for conversions

441b69d

Merge branch 'main' into distributed

3440233

alex-dixon commented Nov 19, 2024

View reviewed changes

src/ell/serialize/http.py Outdated Show resolved Hide resolved

alex-dixon commented Nov 19, 2024

View reviewed changes

src/ell/serialize/http.py Outdated Show resolved Hide resolved

alex-dixon added 6 commits November 19, 2024 07:40

refactor: dedupe http error handling

3db907c

refactor(serialize): put uses in WriteLMPInput instead of second argu…

b1a2445

…ment

add http client tests

1c3af0f

Merge branch 'main' into distributed

121de02

use stores.models

a8e2417

allow dburl none if engine provided

b2b416a

alex-dixon commented Nov 20, 2024

View reviewed changes

Merge branch 'main' into distributed

7bc69f8

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ell serialization refactor / distributed 2 #362

ell serialization refactor / distributed 2 #362

alex-dixon commented Nov 2, 2024 •

edited

Loading

alex-dixon Nov 17, 2024

alex-dixon Nov 17, 2024

alex-dixon Nov 17, 2024

alex-dixon Nov 19, 2024

alex-dixon Nov 19, 2024

alex-dixon Nov 19, 2024 •

edited

Loading

alex-dixon Nov 19, 2024

alex-dixon Nov 20, 2024

alex-dixon Nov 20, 2024



		# Load the image using PIL
		big_picture = Image.open(os.path.join(os.path.dirname(__file__), "bigpicture.jpg"))

-                raise ValueError(f"Parameter {param_name} has no type annotation, and cannot be converted into a tool schema for OpenAI and other providers. Should OpenAI produce a string or an integer, etc, for this parameter?")
+                raise ValueError(f"The {tool_name} tool parameter {param_name} needs a type annotation.
+                Example:
+                def {tool_name}({param_name}: str):
+                                                                       ^^^ Add the type of {param_name} here
+                    Tool function annotations help ell create a tool schema for OpenAI and other providers.")

		Subscriber = WebSocket


		class PubSub(ABC):


		MAX_TOPIC_LENGTH = 65535

		class TopicMatcher:

ell serialization refactor / distributed 2 #362

Are you sure you want to change the base?

ell serialization refactor / distributed 2 #362

Conversation

alex-dixon commented Nov 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alex-dixon Nov 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alex-dixon commented Nov 2, 2024 •

edited

Loading

alex-dixon Nov 19, 2024 •

edited

Loading