[develop2] Running conan install multiple times to support CMake superbuild / FetchContent workflows

[simfinite]

Hi,
I have a set of projects that heavily rely on CMake's FetchContent module to compose what is sometimes referred to as "superbuilds", i.e. I can retrieve projects on-the-fly and add them to the CMake source tree or I can have them checked out side-by-side in a workspace and have CMake traverse the source directories.

In the past, I have used the Conan v1 CMake integration quite successfully here by calling Conan from CMakeList.txt files of all projects that have external dependencies that could be retrieved via Conan (thanks for the effort put into this project, btw).

With the new approach for Conan v2, conan install is only called exactly once on the root source directory. This seems to be incompatible with my old approach. However, I am wondering if this restriction can be lifted? In fact, I hacked a bit into the conan_provider.cmake script and could get a basic example to work by calling conan install once from each CMAKE_CURRENT_LIST_DIR that is seen by the dependency provider callback macro.

I would appreciate any feedback on the possibility of supporting such a workflow. I would be happy to contribute if this is something worth pursuing.

On a side note: I am aware that another solution would be to just use Conan to resolve dependencies everywhere instead of relying on CMake's FetchContent, however, with the above mentioned projects I would like to stick to a "CMake-first workflow" at least for the near future.

[simfinite]

Investigating further, I stumbled upon a problem regarding values of CMAKE_MSVC_RUNTIME_LIBRARY with this approach that was also discussed here #613:

When building a project with multiple dependencies that are included into the build via FetchContent, the top-level project must include the Conan dependency provider script via CMAKE_PROJECT_TOP_LEVEL_INCLUDES. Consequently, when actually building a Conan package for this top-level project, the dependency provider script must also be passed to the CMake build from the recipe, e.g.:

    def build(self):
        cmake = CMake(self)
        cmake.configure(
            variables={"CMAKE_PROJECT_TOP_LEVEL_INCLUDES": "../conan_provider.cmake"}
        )
        cmake.build()

This actually works quite well, however, the toolchain file generated by CMakeToolchain generator will be used and - as discussed in #614 - this may set values for CMAKE_MSVC_RUNTIME_LIBRARY that are currently not supported by conan_provider.cmake. In particular, I could see these additional values in my builds, but there may be more:

$<$<CONFIG:Release>:MultiThreadedDLL>
$<$<CONFIG:Debug>:MultiThreadedDebugDLL>
$<$<CONFIG:Release>:MultiThreadedDLL>$<$<CONFIG:Debug>:MultiThreadedDebugDLL>

[vincent-hui]

I have encountered the same problem. I need to manage a large projects that contains many sub projects.
@simfinite would you mind letting me know whether you use Conan v2 to install dependencies currently? How do you use a single command to install all of the dependencies required in many conanfiles ?

[memsharded]

Hi @simfinite

cmake.configure(
            variables={"CMAKE_PROJECT_TOP_LEVEL_INCLUDES": "../conan_provider.cmake"}
        )

This would be problematic. It is explicitly documented in https://docs.conan.io/2/knowledge/guidelines.html that Conan is not re-entrant, it is not allowed to run Conan inside an already Conan execution. And passing the conan_provider to cmake as variable inside a Conan recipe build() method will implement exactly that, calling conan install inside an already conan ... command. In any case shouldn't it be the CMakeLists.txt code the one that is passing the provider?

@simfinite would you mind letting me know whether you use Conan v2 to install dependencies currently? How do you use a single command to install all of the dependencies required in many conanfiles ?

Just to clarify, having a project that contains subdirectories (lets say add_subdirectory() for example) and each subdirectory has its own conanfile can easily lead to dependencies issues, because the dependencies versions can easily be out of sync, as they are decoupled one from each other. So if one subdirectory implements lib1 which conanfile depends on somelib/1.0 and another subdirectory implements lib2 which conanfile depends on somelib/2.0, some application linking both lib1 and lib2 will have linking errors or even worse, runtime errors.

The recommendation is to have 1 conanfile that defines a jointly compatible set of dependencies versions for the whole project, then each subdirectory will have its own find_package() to define the libs they want to link. But having 1 conanfile will avoid all possible issues.

So it is not very clear the use case for having those different conanfiles in each subdirectory.

If we are talking about the possibility of working on several Conan packages simultaneously at the user level (in general the unit of development is 1 package, not many, as packages have their own version, which means they are independent from each others, and developing 1 package would be the recommended approach), then it is possible that the way to go is the incubating "Workspaces", see https://docs.conan.io/2/incubating.html. The goal of that is to be able to define super projects for different packages (but that is very challenging due to CMake limitations to define it).

[simfinite]

Hi @memsharded, thanks for the feedback. There are a few things to sort out...

The fact that Conan is not re-entrant, I did not know, so thank you for pointing that out. I may have not explained very well, what I tried to do there, so let me re-iterate:

My situation is that I currently have a CMake-first workflow with a sort of source-level dependency management done in a "modern CMake" way, i.e. composing superbuilds where sources are pulled into the build tree using the FetchContent module. Now, to incrementally adapt a Conan workflow, I want to retrieve some dependencies via Conan and feed them into the CMake-based build workflow. This is what cmake-conan is great for and by being able to call conan install multiple times in different sub-projects (this is the key change, I propose here), the migration process is very transparent (i.e. no changes in upper-level projects if lower-level projects are migrated to Conan except for the top-level include). However, when I now want to build a Conan package from one of my libraries, this is where I would have to call Conan in the disallowed re-entrant way which you cited above, because dependencies may have to run conan install to retrieve some of their dependencies.

I think it's very unfortunate that Conan does not allow for this type of "re-entrant execution". Essentially, doesn't this mean that projects relying on the new cmake-conan dependency provider script, generally cannot be built and packaged with Conan?

Having seemingly successfully (i.e. no undefined behavior to be seen yet) integrated both tools, CMake and Conan, in a fully transparent and flexible way as described above, I think that would be very unfortunate.

Regarding version conflicts and other dependency management issues, I understand all the goodness that comes with Conan in comparison to using CMake-based workflows and that's where we are heading. Still, we have to start somewhere and CMake does have some dependency management facilities that we have successfully leveraged in the past.

Regarding the workspaces idea, I am aware and following (actually have been for Conan 1 workspaces), but I agree that mostly working on 1 package at a time is a reasonable approach.

[vincent-hui]

Hi @memsharded , thank you so much for your feedback. You mentioned

So it is not very clear the use case for having those different conanfiles in each subdirectory.

The use case for having those different conanfiles in each subdirectory is managing multiple packages like Colcon. https://blog.conan.io/2024/11/28/Enhancing-ros-builds-with-Conan.html

In other words, I would like to use Conan workspaces instead of Colcon to manage a large scale project.