Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TOKENIZERS_PARALLELISM #312

Open
fstairu opened this issue Jan 26, 2025 · 1 comment
Open

TOKENIZERS_PARALLELISM #312

fstairu opened this issue Jan 26, 2025 · 1 comment

Comments

@fstairu
Copy link

fstairu commented Jan 26, 2025

I'm trying to run FluxGym on Runpod and constantly experience the issues. The last, and guess the least one (!) is
Avoid using tokenizers before the fork if possible
[2025-01-26 12:10:31] [INFO] - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
And that's the training has not yet been started, just the pre-sample was generated ("save_at_first" enabled on advanced setup)
Absolutely have no clue how to fix it, and have no any desire to spent the money for trying this thing fly, just praying it will not resemble into any other issues!

@Racaya
Copy link

Racaya commented Jan 27, 2025

You can either wait a few minutes and the script should continue (at it always did for me).

Or, you can add the environment variable at the top of app.py, where the other env variables are defined:
os.environ['TOKENIZERS_PARALLELISM'] = 'False'

This resolved the problem for me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants