More comprehensive project references/links in 6.x

[ghost]

I've been off using CMake for a while and I miss Premake for my project because frankly I find the CMake syntax a little too awkward. However there are a few features I've used in CMake that I'd like to see if we could support in 6.x and it's really down to having better relationships for projects.

So in CMake when you add include directories you can specify them to be public like this:

target_include_directories(mylib PUBLIC include)

Which not only adds the directory to mylib's include paths, but then if another target links mylib it also gets that include directory automatically. This is a huge time saver for incorporating third party libraries since I can simply add_subdirectory to include their CMakeLists.txt and then add the library to my target's links and all the include directories are set up.

In Premake land we'd have to think about how one would specify such a thing. I don't really like the way CMake mixes the include directories with a "scope" like public/private but it also is nice because there isn't a second list of include directories to maintain. For me personally I'd probably lean towards a separate function so a library project might have:

includeDirs {
   'include',
   'vendor/whatever/include',
}
exportedIncludeDirs {
  'include',
}

Naming not the point but the project is now in control of specifying all the include directories that a project referencing it would need.

The other piece I really like, which is only available on Windows, is the ability to copy all the DLLs referenced by my target to the output directory:

add_custom_command(TARGET polychromatic POST_BUILD
    COMMAND ${CMAKE_COMMAND} -E copy $<TARGET_RUNTIME_DLLS:mygame> $<TARGET_FILE_DIR:mygame>
    COMMAND_EXPAND_LISTS
)

This syntax is really ugly IMO but functionally it's really great. Again I just add projects to my targets links and if they produce a DLL it'll get copied for me. I don't have to do anything manual to get all the DLLs I need in my output directory.

In this case it feels like something that still needs an API but is a little easier since Premake knows if something is a shared library or not and therefore could easily wire up a copy step. Could be as easy as copyLinkedSharedLibsToOutput 'on' (not a great function name but you get the idea anyway).

Curious if anyone more core to the project has done thinking around this type of thing. Is there any interest in this kind of project relationship? Is there a reason why adding this isn't a good idea?

[fsfod]

It sounds like what you describing is what the uses/usage proposals have been aiming for.

[ghost]

Kind of, though that feels both more generic and more involved. IMO that feels like it's more solving the "how do I share configuration details" rather than "how do I get a complete project reference." While I see that you can solve the latter with the former, it doesn't feel as simple. It also requires that library authors split their library definitions explicitly into blocks which IMO hurts the readability of the project.

[nickclark2016]

Completely agree that this is something that should be aimed for in 6.x. I'm not sure how I feel about having a separate list of include directories for exported, as it's a place where users (or implementers) could easily mess up having to make sure the paths are the same. I do think having the public/private inline is something worth striving for. Don't be afraid to break APIs, as this is a new major revision.

[ghost]

Definitely not afraid to break APIs, but just more concerned about the ergonomics of it. However I think if we allow folks to continue to specify includedirs without scopes (since not every project is a library) we could probably find a way to make it nicer. Here's a sketch of what that might look like.

project('MyLib', function()
  files { 'src/mylib.c' }

  -- For shared libraries, links should link the libraries as normal with
  -- nothing bubbling up to the project referencing the library.
  --
  -- For static libraries, links should bubble up to the project referencing
  -- the library. This lets static libs specify their dependencies even though
  -- they can't directly link themselves.
  links {
    'opengl32',
  }

  -- libDirs should follow the same shared vs static convention as links so
  -- that projects referencing static libraries can automatically have the
  -- right libDirs configured for the necessary links.
  libDirs { }

  -- includeDirs is now a table of tables where each one specifies the
  -- "visibility" of the include directory.
  --
  -- public include dirs are automatically referenced by projects that
  -- reference this one.
  --
  -- Private include dirs are only referenced by this library
  --
  -- Maybe for simplicity of folks writing apps (where public/private doesn't
  -- make sense) we allow additional paths separately. These would implicitly
  -- be private.
  includeDirs {
    public = {
      'include',
    },

    private = {
      'private/include1',
      'private/include2',
    },

    'private/include3',
  }
end)

project('MyApp', function() 
  -- Continue to use links for linking projects but now it automatically
  -- handles the links, libDirs, and includeDirs from the referenced project.
  links { 'MyLib' }

  -- Auto-generate build commands to copy shared libraries to the output
  -- directory of MyApp. This allows us to build each project to separate
  -- output directories while ensuring MyApp has the DLLs/dylibs required to
  -- run.
  copyRuntimeDependencies 'On'
end)

We'd need to think about what other properties need to either be inherited or bubble up, or other ways in which things need to be wired up for successful project references. Some things that immediately jump to mind as maybe things to think about are:

• Post-build commands that copy files for libraries. If copyRuntimeDependencies is on we'd likely also want to copy those files to the app's output directory, but currently we don't have a structured way to express that I don't think which would make that hard.
• Preprocessor defines. While generally I don't think it's good form for libraries to have a lot of preprocessor symbols required to include their public headers, we could choose to have defines support the same public vs private notation like includeDirs above.

[nickclark2016]

I'm definitely a fan of the table approach, personally, as well as static libraries having their dependencies bubble up to other consuming projects. One of the things that is/was being discussed is to split the linking into static and shared libraries. Previously, we had just toyed with the idea of linksStatic and linksShared internally. However, I think that something what you have with includeDirs would make sense if we wanted to go that route:

links {
    static = {
        "externalLib"
    },
    shared = {
        "externalDll"
    }
}

I'd also like to split off project dependencies from "links" to just rely on "dependson", inferring the linkage type from the project being consumed.

[ghost]

I am 100% on board with moving project references off of links to keep things more clear. dependson is an interesting one but given it's in 5.x as a non-linking API it feels like it might be confusing for the same named function to suddenly be a linking one in 6.x. It also isn't 100% obvious that dependencies should always link. Maybe a new references or projectReferences function to specify project level references that should be linked?

What's the value of explicitly splitting up links in that way? Just to aid in inferring extensions? If we're splitting project references off such that links is only for non-project references and we're concerned about extensions, perhaps just require the caller to specify them?

links {
  'externalLib.lib',
  'externalDll.dll',
} 

While this does mean you can't share the links across Windows<->Linux/macOS, in my experience that's rarely a big deal or practical, given other naming convention issues (e.g. some libraries make a Lib.dll on Windows but then Lib-2.0.0.dylib for other platforms where the name is no longer the same, regardless of extension).

[nickclark2016]

It's pretty much the extension/linkage type inference issue that you've mentioned. Currently, we have this type of support for Makefiles to explicitly set the linkage type:

links {
    'external:static'
}

To me, that feels a little clunky and would be better solved with a table approach.

As for dependson, I'd be perfectly happy with another name. If we drop that API name, I'd argue for complete removal of dependson. I'm a little leery of "references", as we need to think about the naming scheme with things like C# and other languages. When I think of references with C#, my mind goes to NuGet package references and such. Would we want them all in the same API, different APIs? (Just spit balling, appreciate the good discussion on this)

[ghost]

It's pretty much the extension/linkage type inference issue that you've mentioned. Currently, we have this type of support for Makefiles to explicitly set the linkage type:

links {
    'external:static'
}

To me, that feels a little clunky and would be better solved with a table approach.

Ah I see. Then yeah I agree having links with tables seems fine as well.

As for dependson, I'd be perfectly happy with another name. If we drop that API name, I'd argue for complete removal of dependson. I'm a little leery of "references", as we need to think about the naming scheme with things like C# and other languages. When I think of references with C#, my mind goes to NuGet package references and such. Would we want them all in the same API, different APIs? (Just spit balling, appreciate the good discussion on this)

TBH I'm not sure where I'd use dependson in practice. I don't know why I'd want a project to depend on another being built but then not reference it. Not that it's not useful, but since I don't know where it is I'm not sure if it'd make sense to drop it.

Also hear you on references. Perhaps dependencies? projectDependencies? projectReferences (this being a little more clear due to the project part)? Not really sure here. I definitely am in favor of the API for "here are project references" being separate from other link/reference/package manager things. This will keep the interface clearer for users while making the backend easier for modules since it's not all interspersed.

I might take a stab at some of the changes we're talking about here just to see what it might look like. I've been meaning to dig into the 6.x code a bit more to see if I can help it along as I'm really excited about Premake 6 being a big breaking cleanup type of release.

[ghost]

I've opened up #1927 with what I have so far to try and get some feedback on the general approach. It's a basic working prototype and there's likely a bit of cleanup left to do, but the rough idea is there and it is working for pulling include dirs up to projects that reference other projects.

Also, I noticed 6.x doesn't have links so I can't easily incorporate those right now without first having the baseline links functionality understood. That said, the new setgroup field type I added should be able to be used for links to support grouping by static and shared.

Happy to discuss further here or in the PR itself.

[nickclark2016]

Appreciate you getting this open. I should have time this coming weekend to delve into your PR.