[Bug]: cuFFT error when running with GPU

[gu-ma]

Contact Details

No response

What happened?

When trying to run with gpu

make run track=test.mp3 gpu=true

I have the following error ⬇️ it might be because of missing cuda and / or torch version? I'm using WSL2 with ubt22.4. But this should not matter with Docker anyway....

In which operating system you found the issue?

Linux

Relevant log output

docker run --rm -i --name=demucs --gpus all -v /home/iart/applications/audio/demucs/input:/data/input -v /home/iart/applications/audio/demucs/output:/data/output -v /home/iart/applications/audio/demucs/models:/data/models xserrat/facebook-demucs:latest python3 -m demucs -n htdemucs --out /data/output   --shifts 1 --overlap 0.25 -j 1 "/data/input/test.mp3"
  0%|                                                                                 | 0.0/146.25 [00:00<?, ?seconds/s]Selected model is a bag of 1 models. You will see that many progress bars per track.
Separated tracks will be stored in /data/output/htdemucs
Separating track /data/input/test.mp3
  0%|                                                                                 | 0.0/146.25 [00:00<?, ?seconds/s]
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/usr/lib/demucs/demucs/__main__.py", line 10, in <module>
    main()
  File "/usr/lib/demucs/demucs/separate.py", line 168, in main
    origin, res = separator.separate_audio_file(track)
  File "/usr/lib/demucs/demucs/api.py", line 308, in separate_audio_file
    return self.separate_tensor(self._load_audio(file), self.samplerate)
  File "/usr/lib/demucs/demucs/api.py", line 271, in separate_tensor
    out = apply_model(
  File "/usr/lib/demucs/demucs/apply.py", line 216, in apply_model
    res = apply_model(sub_model, mix, **kwargs, callback_arg=callback_arg)
  File "/usr/lib/demucs/demucs/apply.py", line 251, in apply_model
    res = apply_model(model, shifted, **kwargs, callback_arg=callback_arg)
  File "/usr/lib/demucs/demucs/apply.py", line 290, in apply_model
    chunk_out = future.result()  # type: th.Tensor
  File "/usr/lib/demucs/demucs/utils.py", line 132, in result
    return self.func(*self.args, **self.kwargs)
  File "/usr/lib/demucs/demucs/apply.py", line 317, in apply_model
    out = model(padded_mix)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/lib/demucs/demucs/htdemucs.py", line 538, in forward
    z = self._spec(mix)
  File "/usr/lib/demucs/demucs/htdemucs.py", line 437, in _spec
    z = spectro(x, nfft, hl)[..., :-1, :]
  File "/usr/lib/demucs/demucs/spec.py", line 17, in spectro
    z = th.stft(x,
  File "/usr/local/lib/python3.10/dist-packages/torch/functional.py", line 632, in stft
    return _VF.stft(input, n_fft, hop_length, win_length, window,  # type: ignore[attr-defined]
RuntimeError: cuFFT error: CUFFT_INTERNAL_ERROR
make: *** [Makefile:54: run] Error 1

[gu-ma]

This fix it for me #46