Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix PretrainedTokenizerFast check #35835

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
+2 −2
Open

Fix PretrainedTokenizerFast check #35835

Changes from 3 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
2eeaef4
Fix the bug in tokenizer.save_pretrained when saving tokenizer_class …
CL-ModelCloud Jan 22, 2025
495729f
Update tokenization_utils_base.py
Qubitium Jan 22, 2025
e90338f
Merge branch 'main' into Fix-Tokenizer-Save-Bug
Qubitium Jan 22, 2025
8028463
Update tokenization_utils_base.py
LRL-ModelCloud Jan 23, 2025
06f9bb9
Merge branch 'main' into Fix-Tokenizer-Save-Bug
Qubitium Jan 23, 2025
9fba4b9
Merge branch 'main' into Fix-Tokenizer-Save-Bug
Qubitium Jan 24, 2025
cb16afb
Update tokenization_utils_base.py
LRL-ModelCloud Jan 24, 2025
edbd83c
Merge branch 'main' into Fix-Tokenizer-Save-Bug
Qubitium Jan 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion src/transformers/tokenization_utils_base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2456,8 +2456,10 @@ def save_pretrained(

# Add tokenizer class to the tokenizer config to be able to reload it with from_pretrained
tokenizer_class = self.__class__.__name__
# import here to prevent circular import error
from .tokenization_utils_fast import PreTrainedTokenizerFast
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would love to avoid this, it's pretty weird

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would love to avoid this, it's pretty weird

Another option is to remove this block of Fast strip code completedly. Is this legacy code still needed now?

Copy link
Contributor

@Qubitium Qubitium Jan 23, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are now using different, clean, method to perform the same fix.

# Remove the Fast at the end unless we have a special `PreTrainedTokenizerFast`
if tokenizer_class.endswith("Fast") and tokenizer_class != "PreTrainedTokenizerFast":
if tokenizer_class.endswith("Fast") and not isinstance(self, PreTrainedTokenizerFast):
tokenizer_class = tokenizer_class[:-4]
tokenizer_config["tokenizer_class"] = tokenizer_class
if getattr(self, "_auto_map", None) is not None:
Expand Down