Code

adding tokenizer configuration#19

Code

Merged

usamoi merged 6 commits intotensorchord:maintensorchord/pg_bestmatch.rs:mainfrom jwnz:add-tokenizer-configurationjwnz/pg_bestmatch.rs:add-tokenizer-configurationCopy head branch name to clipboardOct 8, 2024

+1,158-135

Commits

Commits on Sep 29, 2024

• adding dynamic tokenizer configuration

  Show description for fa7aa01
  jwnz

  committed
  fa7aa01

  Copy full SHA for fa7aa01

• adding whitespace and japanese tokenizers, and tests

  Show description for d2ffee1
  jwnz

  committed
  d2ffee1

  Copy full SHA for d2ffee1

• Allow switching of hf tokenizers

  Show description for 1c9a8c7
  jwnz

  committed
  1c9a8c7

  Copy full SHA for 1c9a8c7

• cleanup & remove uneccesary match in tokenize method

  Show description for c06f2ab
  jwnz

  committed
  c06f2ab

  Copy full SHA for c06f2ab

• adding tiktoken tokenizer

  Show description for a6c9462
  jwnz

  committed
  a6c9462

  Copy full SHA for a6c9462

• Formatting Cargo.toml & updating Cargo.lock to match

  Show description for 277fdc7
  jwnz

  committed
  277fdc7

  Copy full SHA for 277fdc7