feat: support dynamic vocab len

[silver-ymz]

close #25

Now all operations about index don't rely on vocab len. Maybe we can remove TOKENIZER_NAME guc, and update tokenizer(text) to tokenizer(text, model_name).

[VoVAllen]

Maybe we can remove TOKENIZER_NAME guc, and update tokenizer(text) to tokenizer(text, model_name).

Looks good to me. Better to use verb like tokenize(text). And do we need to change the syntax to

SELECT id, passage, embedding <&> to_bm25query('documents_embedding_bm25', tokenize('PostgreSQL')) AS rank
FROM documents
ORDER BY rank
LIMIT 10;

or SELECT id, passage, embedding <&> to_bm25query('documents_embedding_bm25', 'PostgreSQL', tokenizer_name) AS rank FROM documents ORDER BY rank LIMIT 10;

[VoVAllen]

And is this related to the posting cursor refactor?

[silver-ymz]

And do we need to change the syntax to

SELECT id, passage, embedding <&> to_bm25query('documents_embedding_bm25', tokenize('PostgreSQL')) AS rank
FROM documents
ORDER BY rank
LIMIT 10;

or SELECT id, passage, embedding <&> to_bm25query('documents_embedding_bm25', 'PostgreSQL', tokenizer_name) AS rank FROM documents ORDER BY rank LIMIT 10;

Yes, Here is an example for updated API:

CREATE TABLE documents (
    id SERIAL PRIMARY KEY,
    passage TEXT
);

INSERT INTO documents (passage) VALUES
('PostgreSQL is a powerful, open-source object-relational database system.');

ALTER TABLE documents ADD COLUMN embedding bm25vector;

UPDATE documents SET embedding = tokenize(passage, "unicode"); -- specify tokenizer here

CREATE INDEX documents_embedding_bm25 ON documents USING bm25 (embedding bm25_ops);

SELECT id, passage, embedding <&> to_bm25query('documents_embedding_bm25', 'PostgreSQL', 'unicode') AS rank -- also specify tokenizer for to_bm25query
FROM documents
ORDER BY rank
LIMIT 10;

And is this related to the posting cursor refactor?

No, it doesn't contain the posting cursor refactor in #18

[VoVAllen]

The updated API looks good to me. Please update the readme accordingly when finished. You can merge it when ready.