Scope out usage of CUDA as cmake language

[bartgol]

Idea: do not rely on nvcc_wrapper for the cuda-vs-host compiler magic, but use native CMake stuff (assuming cmake version is 3.18 or higher). Possible approach: set a var CXX_LANG which evaluates to CXX for non-cuda builds, and to CUDA for cuda builds. Then manually set the language of all src files to ${CXX_LANG}, to force compilation with the proper compiler.

I speculate that this would also allow to get stuff like

target_compile_features(mylib PRIVATE cxx_constexpr)

to work correctly. I think the nvcc_wrapper layer is somehow blocking communication between the compiler and cmake, preventing correct detection of cxx features. By using native CMake support for CUDA, I suspect this could be fixed, since cmake would invoke nvcc for compiling CUDA sources, folllowed by an invoke of the host compiler on the generated intermediate cpp source.

[bartgol]

E.g., do something like this:

...
if (Kokkos_ENABLE_CUDA)
  set(Kokkos_LANG CUDA)
else ()
  set (Kokkos_LANG CXX)
endif()
project (ekat C Fortran ${Kokkos_LANG})
...

add_library(ekat ${EKAT_SOURCES})
set_source_files_properties (${EKAT_SOURCES} PROPERTIES LANGUAGE ${Kokkos_LANG})

This should a) be made flexible so to only set the language for cxx sources (do a foreach over src files, check file language, if cxx, then override with ${Kokkos_LANG}), and b) provide macros for downstream apps to create libs/execs, and do this language setting magic inside these macros. Something like

macro(ekat_add_library libName ...)
  add_library(${libName} ...)
  # set LANGUAGE=${Kokkos_LANG} for all CXX sources
endmacro()