feat: Add contraints and embedding directives

[fredcarle]

Relevant issue(s)

Resolves #3350
Resolves #3351

Description

Sorry for having 3 technically separate features as part of 1 PR. The reason for this is that I started working on embeddings and the size contraint was initially part of the embedding. Since we discussed that it could be applied to all array fields (and later even to String and Blob), I extracted it into a contraints directive that has a size parameter (more contraints can be added in the future). Furthermore, embeddings returned from ML models are arrays of float32. This caused some precision issues because we only supported float64. When saving the float32 array, querying it would return an float64 array with slight precision issues. I decided to add the float32 type.

You can review the first commit for contraint and embedding related code and the second commit for the float related changes. Some float stuff might have leaked in the first commit. Sorry for this. I tried hard to separate the float32 related changes.

Note that the gql.Float type is now Float64 internally.

type User {
  points: Float
}

is the same as

type User {
  points: Float64
}

The embedding generation relies on a 3rd party package called chromem-go to call the model provider API. As long as one of the supported provider API is configured and accessible, the embeddings will be generated when adding new documents. I've added a step in the test workflow that will run the embedding specific tests on linux only (this is because installation on windows and mac is less straight forward) using Ollama (because it runs locally).

The call to the API has to be done synchronously otherwise the docID/CID won't be representative of the contents. The only alternative would be for the system to automatically update the document when returning from the API call but I see that as a inferior option as it hides the update step from the user. It could also make doc anchoring more complicated as the user would have to remember to wait on the doc update before anchoring the doc at the right CID.

We could avoid having embedding generation support and let the users do that call themselves and store the embedding vectors directly. However, having it as a feature allows us to support RAG vector search which would let users get started with AI with very little work. This seems to be something our partners are looking forward to.

I don't see the 3rd party API call inline with a mutation as a problem since this is something that has to be configured by users and users will expect the mutation calls to take more time as a result.

If you're interested in running it locally, install Ollama and define a schema like so

type User {
    name: String
    about: String
    name_v: [Float32!] @constraints(size: 768) @embedding(fields: ["name", "about"], provider: "ollama", model: "nomic-embed-text",  url: "http://localhost:11434/api") // contraint is optional and localhost:11434 is the default port for Ollama
}

Next steps:

• Support templates for the content sent to the model.
• Add the _similarity operation to calculate the cosine similarity between two arrays.

Tasks

• I made sure the code is well commented, particularly hard-to-understand areas.
• I made sure the repository-held documentation is changed accordingly.
• I made sure the pull request title adheres to the conventional commit style (the subset used in the project can be found in tools/configs/chglog/config.yml).
• I made sure to discuss its limitations such as threats to validity, vulnerability to mistake and misuse, robustness to invalidation of assumptions, resource requirements, ...

How has this been tested?

make test and manual testing

Specify the platform(s) on which this was tested:

• MacOS

[AndrewSisley]

question: This is making a remote call to a third party service no (ollama or openai)?

[fredcarle]

If embedding is defined yes.

[AndrewSisley]

That sounds discussion-worthy 😁

todo: Please significantly expand the documentation on the Embedding related types/properties in client, so both ourselves and users are very aware that this behaviour exists.

todo: Please describe to the team why we really need to do this (on doc create/update) in the PR description, including what we are looking to gain from the call (where/when-ever it is made from), how long you expect it to remain in it's current state (e.g. synchronous, on doc create/update), and any alternative and long-term thoughts you may have on it.

todo: Please flag this in discord, or perhaps the standup (if everyone is going to be present), as I think the team needs to be aware of this - making mandatory calls (depending on field kind etc) to third party APIs on document creation and update from a database does sound a little unusual and does appear to me to come with significant downsides.

[fredcarle]

PR description has been updated

[jsimnz]

Follow up: Discussed during the standup. We'll leave the "in-request" embedding network call as is for now, and start expanding other options for async embedding generation soon.

cc: @fredcarle can you create a follow up ticket for this?

[AndrewSisley]

I think the calls out to 3rd party APIs needs expanded documentation and discussion with the team. I have reviewed very little so far but will continue whilst that issue is being resolved.

[shahzadlone]

question: This is only needed for basic test matrix?

[fredcarle]

Yes. Why are you asking?

[shahzadlone]

because:

1. I don't understand why this is not needed for other test types (acp, view, encryption, etc.).
2. Is this a required dependency now? if so then:
   • document in README
   • add a make rule in Makefile
   • perhaps add this to the setup composite instead (./.github/composites/setup-defradb)

[fredcarle]

1. When you see "Ollama", do you need more documentation to know what this is about? I might be wrong to assume that would be enough to understand that this is only for embedding related testing.
2. It's not a Defra dependency. Only a testing dependency for the embedding specific tests.

[shahzadlone]

1. Maybe I missed it but last I checked the embedding tests they didn't have SupportedTypes like config. So I assumed they will run with all configurations. For example when encryption or macos tests run, wouldn't they still run and hence be need this dep?

[shahzadlone]

2. Yea so similar to how we have rust dep for lens test, this should have a make rule imo and will be part of test/dev deps

[AndrewSisley]

I've given the production code a first-pass review, and will shortly look through the tests. The code looks good, most of the comments are about moving stuff around, or questions/thoughts RE the feature design.

thought: It looks very tedious and error prone to introduce new types atm, although numerics are probably most painful. Perhaps we should review certain aspects in the future to improve this.

[AndrewSisley]

question: This error only appears to be returned when getting the value, I thought we had a dedicated space for validating doc contents - would it be more appropriate to handle this there? Or at least on doc-value write (maybe in addition-too read)?

[fredcarle]

This function is called by validateFieldSchema. So it's done on write.

[AndrewSisley]

Nice, thanks!

[AndrewSisley]

question: It looks like if a user explicitly provides their own vector value it will be overwritten by the result of the remote call. Would it not be better to allow users to provide their own values?

Besides being less surprising for them, it would give them greater control over the determinism of the resultant CID/DocID, which I believe can vary depending on the timing of the remote calls (under most AI models).

[fredcarle]

It shouldn't overwrite. If it does it's a mistake on my part. Users should be free to use their own vectors.

[AndrewSisley]

Ah okay nice!

todo: I've not read the tests yet, could you please make sure that there is at least one that covers that scenario?

suggestion: Please also note this behaviour in the func documentation

[AndrewSisley]

suggestion: Inverting this if and continue-ing would remove quite a long indentation, and IMO make the code a bit easier to read.

[jsimnz]

related to my suggestion above regarding moving the EmbeddingDescription to the CollectionDescription instead of the CollectionFieldDescription would simplify this function, where it can go straight to the embeddings rather than scanning all the fields to see if there is a description on them.

[AndrewSisley]

suggestion: StringBuilder might be more appropriate here, it might be more readable, and should be more efficient, although the performance gains would probably be small compared to the cost of a remote call.

[AndrewSisley]

todo: This error should be declared in an errors.go file

[AndrewSisley]

todo: There appears to be a lot of schema/collection description validation taking place in this file. Please remove as much as you can and instead place it in internal/db/definition_validation.go.

This will allow all our schema validation to live in once place, and lets it operates on our lowest-level/core types, instead of within query-language dependent code-spaces.

[jsimnz]

are you referring to the new code in this PR or in general for this file?

[AndrewSisley]

The new code in this PR. We have existing validation in this file, but only for GQL/SDL specific validation. The current PR contains a lot of general validation, such as validating that target is an array-float kind - this needs to move.

[AndrewSisley]

todo: Please move these descriptions to the descriptions.go file

[AndrewSisley]

todo: Please move this description to descriptions.go

[AndrewSisley]

I've reviewed the tests, thanks for adding them, I think there are a couple of important test gaps though, please add tests for:

1. Patching a size constraint, including removing it
2. Patching an embedding property, and it's children:
   2.1 Adding an embedding description to a field that didnt have one before
   2.2 Mutating each of the embedding properties on a field that declared one on create
   2.3 Removing an embedding property from a field that had one on create
3. Including a size constraint in a View declaration, if you don't/haven't blocked it off, please include a test that shows what happens if that constraint is violated
4. Including an embedding property in a View declaration (I assume we want this to error for now).

[AndrewSisley]

suggestion: _docID is quite an expensive property maintenance-wise to include in a test, and it does not appear to have a use here, I suggest removing it from the query.

Same goes for the p[n]counter tests.

[AndrewSisley]

todo: Please document in this file why these tests are Linux only. I know you noted this in the PR description, but if this constraint persists it will take some figuring out as to why it is Linux only.

suggestion: Consider creating a ticket and linking it to and from here to remove this constraint.

[AndrewSisley]

thought: I'm not sure including the full array helps people read the test, and the only benefit to asserting anything beyond 'is it an array with some values' is perhaps the change detector.

question: Do you have plans to introduce more tests like this in the near future?

suggestion: If you do want to add more, can we perhaps look at asserting on this value in a slightly different way, perhaps only checking that it is of an expected size, with non-zero values.

[islamaliev]

as an idea, we can create a new Validator for asserting this. Similar to NewUniqueCid in tests/integration/results.go

[AndrewSisley]

todo: Please change the way you constrain these tests. The current setup will prevent me, or any other linux users from running go test ./... or even make test, this could be a bit of a nuisance. Perhaps introduce something similar to the SupportedClients TestCase property.

[shahzadlone]

question: curious why this needed to be done?

[fredcarle]

Because the ids were already available on create and since the embedding vector is not know in advance, we cannot depend on the documents content in the test's definition to determine the docIDs.

[jsimnz]

Marking this as "request for changes" (you can dismiss at any time), even tho there aren't any explcit todos.

I have mainly focused on the definition/description code along with the collection/doc updating. Speed ran through the encoding section and the gql parsing.

Haven't reviewed the tests.

Main question/discussion points are "when" you are generating the embedding in the call path, and the location of the EmbeddingDescription.

[jsimnz]

suggestion: What if the structure/placement of the EmbeddingDescription was similar to how we have IndexDescription, where it actually lives on the CollectionDescription instead here on the CollectionFieldDescription.

This would make it much easier to iterate through the existing embedding descriptions when mutating documents to update/generate the embeddings. Whereas right now you have to iterate through all the fields on a doc.

It also opens up the door to not having to actually have the embedding in the document graph itself, and instead is stored as an index outside the graph (future work).

[jsimnz]

question: Is there a particular reason why the first thing the create/update call does is generate the embeddings? in the update case, it gets the embeddings before the ACP check is even done to determine if a mutation is allowed? It could even be in the save func deeper in the call graph instead of duplicated in the create and update paths.

[jsimnz]

related to my suggestion above regarding moving the EmbeddingDescription to the CollectionDescription instead of the CollectionFieldDescription would simplify this function, where it can go straight to the embeddings rather than scanning all the fields to see if there is a description on them.

[jsimnz]

question: Should there be an additional check here if we are creating a new doc? Since we can resolve the missingFieldsForGenerating by getting the old doc version if this is a create call. Or is this covered by the c.get call returning an error not found?

[jsimnz]

suggestion: might want to wrap this error so its clear to the caller why a document is trying to be fetched.

[jsimnz]

question: I cant remember where we are it regarding dev policies for inline error creation? Does this need to be a properly defined error in the errors.go file?

[AndrewSisley]

Yes please :)

[jsimnz]

suggestion: wrap this error

[jsimnz]

are you referring to the new code in this PR or in general for this file?

[islamaliev]

nitpick: I don't see a point of having the private function

[islamaliev]

todo: add please tests for float32

[islamaliev]

question: why not to make them 8 and 9 so that they are next to their siblings?

[islamaliev]

todo: please cover float32 as well

[islamaliev]

as an idea, we can create a new Validator for asserting this. Similar to NewUniqueCid in tests/integration/results.go

[islamaliev]

suggestion: I would move this part on the GraphQL side to catch error earlier and potentially make better user experience.

It should work pretty much the same way as we check fields for indexes. Find in the project indexFieldFromAST.

I would make it "todo" actually, but our experiment... :)

[fredcarle]

Not sure what you mean by move to the GQL side. This is the generation part so the embedding on the GQL side isn't really involved prior to this.