tokenizer issue

[imsamimalik]

Hi! I am getting this issue after the installing the dependencies
RuntimeError: Failed to import transformers.models.auto because of the following error (look up to see its traceback): No module named 'tokenizers.pre_tokenizers'

Is anyone else facing this issue?

[caochengchen]

me too

[caochengchen]

Let the author solve the problem

[Dashvvood]

I believe this may be caused by a naming conflict between the "tokenizers" module in the project directory and the "huggingface/tokenizers" module. I attempted to resolve it by renaming the "tokenizers" folder in the project to "ham2pose_tokenizers" and modifying the import section of the "train.py" file. After these changes, it should run successfully.

[caochengchen]

我相信这可能是由于项目目录中的“tokenizers”模块和“huggingface/tokenizers”模块之间的命名冲突引起的。我试图通过将项目中的“tokenizers”文件夹重命名为“ham2pose_tokenizers”并修改“train.py”文件的导入部分来解决此问题。完成这些更改后，它应该会成功运行。

Did you run it successfully？

[imsamimalik]

I believe this may be caused by a naming conflict between the "tokenizers" module in the project directory and the "huggingface/tokenizers" module. I attempted to resolve it by renaming the "tokenizers" folder in the project to "ham2pose_tokenizers" and modifying the import section of the "train.py" file. After these changes, it should run successfully.

Yes, that's how I fixed it.