Refactored modules/tokenizers to be a subdir of modules/transforms

[Ankur-singh]

Context

What is the purpose of this PR? Is it to

• add a new feature
• fix a bug
• update tests and/or documentation
• other (Refactored to convey that tokenizers are a subset of transforms.)

Please link to any issues this PR addresses. #1301

Changelog

What are the changes made in this PR?

• Moved modules/tokenizers to be a subdir of modules/transforms
• Updated corresponding imports and doc strings
• Updated Docs and links to source code

Test plan

Please make sure to do each of the following if applicable to your PR. If you're unsure about any one of these just ask and we will happily help. We also have a contributing page for some guidance on contributing.

• run pre-commit hooks and linters (make sure you've first installed via pre-commit install)
• add unit tests for any new functionality
• update docstrings for any new or updated methods or classes
• run unit tests via pytest tests
• run recipe tests via pytest tests -m integration_test
• manually run any new or modified recipes with sufficient proof of correctness
• include relevant commands and any other artifacts in this summary (pastes of loss curves, eval results, etc.)

(tune) ➜  torchtune git:(move-tokenizers-to-transforms) ✗ tune run eleuther_eval --config recipes/configs/qwen2_5/evaluation.yaml         
Running EleutherEvalRecipe with resolved config:

batch_size: 8
checkpointer:
  _component_: torchtune.training.FullModelHFCheckpointer
  checkpoint_dir: /tmp/Qwen2_5-0_5B-Instruct
  checkpoint_files:
  - model.safetensors
  model_type: QWEN2
  output_dir: ./
device: cuda
dtype: bf16
enable_kv_cache: true
limit: null
max_seq_length: 4096
model:
  _component_: torchtune.models.qwen2_5.qwen2_5_0_5b
output_dir: ./
quantizer: null
seed: 1234
tasks:
- truthfulqa_mc2
tokenizer:
  _component_: torchtune.models.qwen2_5.qwen2_5_tokenizer
  max_seq_len: null
  merges_file: /tmp/Qwen2_5-0_5B-Instruct/merges.txt
  path: /tmp/Qwen2_5-0_5B-Instruct/vocab.json

2025-01-05:10:51:01,626 INFO     [_utils.py:28] Running EleutherEvalRecipe with resolved config:

batch_size: 8
checkpointer:
  _component_: torchtune.training.FullModelHFCheckpointer
  checkpoint_dir: /tmp/Qwen2_5-0_5B-Instruct
  checkpoint_files:
  - model.safetensors
  model_type: QWEN2
  output_dir: ./
device: cuda
dtype: bf16
enable_kv_cache: true
limit: null
max_seq_length: 4096
model:
  _component_: torchtune.models.qwen2_5.qwen2_5_0_5b
output_dir: ./
quantizer: null
seed: 1234
tasks:
- truthfulqa_mc2
tokenizer:
  _component_: torchtune.models.qwen2_5.qwen2_5_tokenizer
  max_seq_len: null
  merges_file: /tmp/Qwen2_5-0_5B-Instruct/merges.txt
  path: /tmp/Qwen2_5-0_5B-Instruct/vocab.json

Model is initialized with precision torch.bfloat16.
2025-01-05:10:51:02,219 INFO     [eleuther_eval.py:503] Model is initialized with precision torch.bfloat16.
2025-01-05:10:51:02,419 INFO     [huggingface.py:132] Using device 'cuda:0'
2025-01-05:10:51:02,886 INFO     [huggingface.py:369] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
Running evaluation on the following tasks: ['truthfulqa_mc2']
2025-01-05:10:51:13,798 INFO     [eleuther_eval.py:540] Running evaluation on the following tasks: ['truthfulqa_mc2']
2025-01-05:10:51:13,799 INFO     [task.py:415] Building contexts for truthfulqa_mc2 on rank 0...
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 817/817 [00:00<00:00, 1745.50it/s]
2025-01-05:10:51:14,294 INFO     [evaluator.py:496] Running loglikelihood requests
Running loglikelihood requests: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5882/5882 [08:40<00:00, 11.31it/s]
Eval completed in 522.65 seconds.
2025-01-05:10:59:56,450 INFO     [eleuther_eval.py:549] Eval completed in 522.65 seconds.
Max memory allocated: 10.00 GB
2025-01-05:10:59:56,451 INFO     [eleuther_eval.py:550] Max memory allocated: 10.00 GB


|    Tasks     |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|--------------|------:|------|-----:|------|---|-----:|---|-----:|
|truthfulqa_mc2|      2|none  |     0|acc   |↑  |0.4178|±  |0.0146|


2025-01-05:10:59:56,536 INFO     [eleuther_eval.py:554] 

|    Tasks     |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|--------------|------:|------|-----:|------|---|-----:|---|-----:|
|truthfulqa_mc2|      2|none  |     0|acc   |↑  |0.4178|±  |0.0146|

UX

If your function changed a public API, please add a dummy example of what the user experience will look like when calling it.
Here is a docstring example
and a tutorial example

• I did not change any public API
• I have added an example to docs or docstrings
• Updated the docs

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/2231

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit aaf416f with merge base 75965d4 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 90.47619% with 4 lines in your changes missing coverage. Please review.

Project coverage is 64.10%. Comparing base (779569e) to head (aaf416f).
Report is 6 commits behind head on main.

Files with missing lines	Patch %	Lines
torchtune/modules/tokenizers/__init__.py	0.00%	3 Missing ⚠️
recipes/eleuther_eval.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2231      +/-   ##
==========================================
- Coverage   65.34%   64.10%   -1.24%     
==========================================
  Files         358      354       -4     
  Lines       21207    20674     -533     
==========================================
- Hits        13857    13254     -603     
- Misses       7350     7420      +70     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Ankur-singh]

@joecummings looks like there is a merge conflict but I am unable to see it. How do I fix it?

[ebsmothers]

@Ankur-singh sorry we missed this comment until now. In my Github UI I see the following just above the "Add a comment" box (might be slightly different depending on your permissions).

So seems like the merge conflict is in _sentencepiece.py

[Ankur-singh]

@ebsmothers fixed merge conflict and successfully ran pytest tests. Should work now.

[ebsmothers]

Thanks @Ankur-singh for the PR! Overall I think the changes make sense, main comment is around backwards compatibility. This PR will definitely break public APIs (e.g. from torchtune.modules.tokenizers import TikTokenBaseTokenizer will break). I think the fix is to keep torchtune/modules/tokenizers/__init__.py with e.g. from torchtune.modules.transforms.tokenizers import TikTokenBaseTokenizer. Then add a deprecation flag announcing the change from torchtune.modules.tokenizers to torchtune.modules.tokenizers.transforms import path. And you can add a comment to torchtune/modules/tokenizers/__init__.py explaining that these imports are for backwards compatibility purposes and will be removed in v0.7. This way people can still import from torchtune.modules.tokenizers in the short term.

[Ankur-singh]

thanks for the feedback @ebsmothers. I'll make the required changes.

This is my first time contributing to a big project (like Torchtune), so I’m still getting the hang of things. In the smaller projects I contributed to before, backward compatibility and versioning weren’t as big of a concern. Please bear with me as I work through any rookie mistakes—I’m already learning a lot and appreciate your guidance!

[ebsmothers]

Thanks @Ankur-singh for sticking with this one! I know big horizontal refactors like this can be a bit of a pain so definitely appreciate your diligence. Just a couple more small comments and I think we're good here. Can you also check docs/source/basics/custom_components.rst and docs/source/basics/model_transforms.rst? I think there's still a modules.tokenizers import path in each of those files.

[Ankur-singh]

@ebsmothers I’m having a blast contributing to Torchtune and learning a ton about engineering best practices along the way. I’ve made the necessary changes! I do tend to miss the occasional small detail, but I’ll be more mindful going forward. Thanks for your patience—I really appreciate it!