Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tensorflow2.4 support #104

Open
luisarandas opened this issue Mar 24, 2021 · 5 comments
Open

Tensorflow2.4 support #104

luisarandas opened this issue Mar 24, 2021 · 5 comments

Comments

@luisarandas
Copy link

Hello.

Is this repository working with TF>2? I'm having a hard time with virtual environments and these older versions on linux.

Thanks

@chrisdonahue
Copy link
Owner

Unfortunately the repo has not yet been updated to work with TF>2. It might be fairly straightforward to patch it using tf.compat.v1, though I don't have much experience with TF2 :(

@luisarandas
Copy link
Author

luisarandas commented Mar 29, 2021

I think I managed a workaround on Linux with tf-gpu 2.2.0.0rc1 if you want I can fork and upload (with tf.compat.v1, which is not super nice but it is training right now). Tho I can't access the tensorboard with the newest cuda drivers.

ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory

Had problems with the device search and added thecommon workaround

sess = tf.compat.v1.Session(config=tf.ConfigProto( allow_soft_placement=True, log_device_placement=True))

Will see how this goes. How much time/iterations do you recommend? Kind of blind here lol
Thank you

@markhanslip
Copy link

Hi @luisarandas,

Did you manage to get this working? It was working fine for me with TF 1.13 middle of last year, but now I'm revisiting it and it no longer works properly. The model is clearly in the GPU memory and it claims to be training, but for some reason it's progressing at CPU-like speeds. The same behaviour is happening locally and in Colab. I don't have such problems with other repos that use TF v2 hence I'm wondering if the tf.compat.v1 workaround works with this codebase. Obviously I will try it but these things take time and it would be great to hear if you have something working already!

Thanks,

Mark

@mattjwarren
Copy link

I have created an updated fork of this repository migrated to run on Tensorflow 2. You may have more success getting it to run in modern environments, especially if you are aiming to train on newer GPU hardware. https://github.com/mattjwarren/wavegan

@markhanslip
Copy link

Thanks for sorting this out, appreciated. It was getting annoying to have to restart Colab every time it gave me an A100

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants