You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on May 17, 2023. It is now read-only.
The Quicktour from Huggingface ends with the following paragraph :
but currently, {hftokenizer} do not allow to load pretrained tokenizers.
expected behavior
I'd like to be able to reuse pretrained tokenizers already available in the LM models present in the wild ( BERT, RoBERTa and friends) and / or in my local cache folder, in order to feed those models with the result of {hftokenizer} tokenizer$encode()$ids.
And have the Quicktour vignette to cover the API to do it.
Thanks a lot !
The text was updated successfully, but these errors were encountered:
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Hello,
current behavior
The
Quicktour
from Huggingface ends with the following paragraph :but currently, {hftokenizer} do not allow to load pretrained tokenizers.
expected behavior
I'd like to be able to reuse pretrained tokenizers already available in the LM models present in the wild ( BERT, RoBERTa and friends) and / or in my local cache folder, in order to feed those models with the result of {hftokenizer}
tokenizer$encode()$ids
.And have the
Quicktour
vignette to cover the API to do it.Thanks a lot !
The text was updated successfully, but these errors were encountered: