BPE tokenizer

[rom1504]

Could be fun to have a tokenizer like "take all video frames, apply clip, transform into cluster N of 2^17 (what I have in clip retrieval index), apply BPE, return sequence"

Inspired by https://arxiv.org/abs/2309.04459

[rom1504]

Can't find any numbers on reduction of sequence length that BPE provides

[rom1504]

But let's say 3. That would mean encoding a video of 3600 frames as a sequence of 1200 tokens