Issue with CUDA 11.7 build

[hmacdope]

Solution to issue cannot be found in the documentation.

• I checked the documentation.

Issue

mamba install openmm cudatoolkit=11.7
python -m openmm.testInstallation

Gives

OpenMM Version: 8.0
Git Revision: a7800059645f4471f4b91c21e742fe5aa4513cda

There are 4 Platforms available:

1 Reference - Successfully computed forces
2 CPU - Successfully computed forces
3 CUDA - Error computing forces with CUDA platform
4 OpenCL - Successfully computed forces

CUDA platform error: Error initializing FFT: 5

Median difference in forces between platforms:

Reference vs. CPU: 6.31628e-06
Reference vs. OpenCL: 6.74399e-06
CPU vs. OpenCL: 7.44203e-07

NVIDIA-smi output on 4090

Sat Oct  7 11:43:54 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.125.06   Driver Version: 525.125.06   CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+

Installing with the default in a fresh env (pulls down cudatoolkit=11.8) seems toi be fine

mamba install openmm
>>>
OpenMM Version: 8.0
Git Revision: a7800059645f4471f4b91c21e742fe5aa4513cda

There are 4 Platforms available:

1 Reference - Successfully computed forces
2 CPU - Successfully computed forces
3 CUDA - Successfully computed forces
4 OpenCL - Successfully computed forces

Median difference in forces between platforms:

Reference vs. CPU: 6.31577e-06
Reference vs. CUDA: 6.73608e-06
CPU vs. CUDA: 7.05123e-07
Reference vs. OpenCL: 6.74399e-06
CPU vs. OpenCL: 7.37505e-07
CUDA vs. OpenCL: 2.12823e-07

All differences are within tolerance.

Installed packages

_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2023.7.22            hbcca054_0    conda-forge
cudatoolkit               11.7.1              h4bc3d14_12    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libblas                   3.9.0           18_linux64_openblas    conda-forge
libcblas                  3.9.0           18_linux64_openblas    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.2.0               h807b86a_2    conda-forge
libgfortran-ng            13.2.0               h69a702a_2    conda-forge
libgfortran5              13.2.0               ha4646dd_2    conda-forge
libgomp                   13.2.0               h807b86a_2    conda-forge
liblapack                 3.9.0           18_linux64_openblas    conda-forge
libnsl                    2.0.0                hd590300_1    conda-forge
libopenblas               0.3.24          pthreads_h413a1c8_0    conda-forge
libsqlite                 3.43.0               h2797004_0    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_2    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
ncurses                   6.4                  hcb278e6_0    conda-forge
numpy                     1.26.0          py311h64a7726_0    conda-forge
ocl-icd                   2.3.1                h7f98852_0    conda-forge
ocl-icd-system            1.0.0                         1    conda-forge
openmm                    8.0.0           py311h28d7ac7_3    conda-forge
openssl                   3.1.3                hd590300_0    conda-forge
pip                       23.2.1             pyhd8ed1ab_0    conda-forge
python                    3.11.6          hab00c5b_0_cpython    conda-forge
python_abi                3.11                    4_cp311    conda-forge
readline                  8.2                  h8228510_1    conda-forge
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
tk                        8.6.13               h2797004_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
wheel                     0.41.2             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge

Environment info

active environment : openmm_cudatk_11.7
    active env location : /home/hmacdope/mambaforge/envs/openmm_cudatk_11.7
            shell level : 3
       user config file : /home/hmacdope/.condarc
 populated config files : /home/hmacdope/mambaforge/.condarc
          conda version : 23.1.0
    conda-build version : not installed
         python version : 3.10.12.final.0
       virtual packages : __archspec=1=x86_64
                          __cuda=12.0=0
                          __glibc=2.35=0
                          __linux=6.2.0=0
                          __unix=0=0
       base environment : /home/hmacdope/mambaforge  (writable)
      conda av data dir : /home/hmacdope/mambaforge/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /home/hmacdope/mambaforge/pkgs
                          /home/hmacdope/.conda/pkgs
       envs directories : /home/hmacdope/mambaforge/envs
                          /home/hmacdope/.conda/envs
               platform : linux-64
             user-agent : conda/23.1.0 requests/2.31.0 CPython/3.10.12 Linux/6.2.0-34-generic ubuntu/22.04.2 glibc/2.35
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

[hmacdope]

Tagging @mikemhenry

[ijpulidos]

@hmacdope I cannot reproduce. I did

mamba create -n openmm8-latest-build-test-cu117 -c conda-forge openmm cudatoolkit=11.7
python -m openmm.testInstallation

output is:

OpenMM Version: 8.0
Git Revision: a7800059645f4471f4b91c21e742fe5aa4513cda

There are 4 Platforms available:

1 Reference - Successfully computed forces
2 CPU - Successfully computed forces
3 CUDA - Successfully computed forces
4 OpenCL - Successfully computed forces

Median difference in forces between platforms:

Reference vs. CPU: 6.30386e-06
Reference vs. CUDA: 6.73781e-06
CPU vs. CUDA: 7.33901e-07
Reference vs. OpenCL: 6.74399e-06
CPU vs. OpenCL: 7.66719e-07
CUDA vs. OpenCL: 2.20058e-07

All differences are within tolerance.

nvidia smi: NVIDIA-SMI 535.104.05 Driver Version: 535.104.05 CUDA Version: 12.2

I wonder if it's something in your local setup/driver. I tried this in three different machines (two local workstations and HPC system), all working as expected.

[hmacdope]

OK thanks for checking, must be something with my local setup / the one we were using on HPC.

[hmacdope]

Could also be interaction with the driver version so I will leave open for now.

[mikemhenry]

@ijpulidos when you get a chance, can you run conda list openmm and paste the output? I just want to make sure that you were using the same build number as @hmacdope (I also cannot reproduce this).