SE-0403: Package Manager Mixed Language Target Support

Evolution Proposal Reviews
1

Hello Swift Community,

The review of SE-0403: Package Manager Mixed Language Target Support begins now and runs through July 28, 2023.

Reviews are an important part of the Swift evolution process. All review feedback should be either on this forum thread or, if you would like to keep your feedback private, DM on the forums. When contacting me directly about this proposal, please put "[SE-0402]" at the start of the subject line.

What goes into a review?

The goal of the review process is to improve the proposal under review through constructive criticism and, eventually, determine the direction of Swift. When writing your review, here are some questions you might want to answer in your review:

  • What is your evaluation of the proposal?
  • Is the problem being addressed significant enough to warrant a change to Swift?
  • Does this proposal fit well with the feel and direction of Swift?
  • If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
  • How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

More information about the Swift evolution process is available at

swift-evolution/process.md at main · apple/swift-evolution · GitHub

Thank you for contributing to Swift.

Saleem Abdulrasool
Review Manager

16 Likes
2
  • General evaluation: Overall this seems like a good thing for SwiftPM to be able to do.
  • Effort: Glanced through most of it, but focused on the Modeling a mixed language target and its build process and Building a mixed language target sections.

I had a few questions that came to mind:

  1. CMake mixed targets:

CMake can handle add_library(foo foo.c foo.swift foo.cpp) to generate mixed targets today. One of the challenges was on the linking side. To match the model that CMake was already using, we decided to generate the object files for the C and C++ sources first, and then pass those to the swiftc invocation so that swiftc would handle scheduling building the Swift files since the , linking the C/C++ objects with the Swift objects as appropriate, and providing the linker with the additional swift-specific search paths and linking swiftrt.o. I haven't set up the header generation yet, so folks have to use a custom command at the moment unfortunately.

As you noted with respect to bridging headers, this has the drawback that you have to run the Swift compiler twice, once to generate a header, and then again to build the Swift files. My question is, since you're running the Swift compiler first, what does the link phase look like?

  1. C++ interop

Additionally, when present, each non-Objective-C/C header (e.g. a C++ header) is excluded from the module. This avoids build errors that arise when exposing, for example, a C++ header to Swift

From the proposal, C++ headers are left out of the module. What is the plan for adding C++ interop support to this?

  1. Specifying flags to clang build job vs Swift build job vs Link step

I think I got a little confused by this portion:

Build flags for the Swift part of the target

The following flags are used when compiling the Swift part of the target:

  1. -import-underlying-module This flag triggers a partial build of the underlying C language sources when building the Swift module. This critical flag enables the Swift sources to use C language types defined in the Clang part of the target.
  2. -I /path/to/overlay_directory The above -import-underlying-module flag will look for a module map in the given header search path. The overlay directory chosen when creating the above VFS overlay files is used here.
  3. -ivfsoverlay /path/to/Intermediates/all-product-headers.yaml This enables the overlay to take effect during the compilation. Specifically, it will be used during the partial build of the C language sources that is triggered by the -import-underlying-module flag.
  4. -ivfsoverlay /path/to/Intermediates/unextended-module-overlay.yaml This enables the overlay to take effect during the compilation. Specifically, it will be used when compiling the Swift sources.
  5. -I $(target’s path) Adding the target's path allows for importing headers using paths relative to the root of the target. Because passing -import-underlying-module triggers a partial build of the C language sources, this is needed for resolving possible header imports.
Build flags for the Clang part of the target

The following flags are used when compiling the Clang part of the target:

  1. -I $(target’s path) Adding the target's path allows for importing headers using paths relative to the root of the target.
  2. -ivfsoverlay /path/to/Intermediates/all-product-headers.yaml This enables the overlay to take effect during the compilation.
  3. -I /path/to/Intermediates/ The above overlay virtually adds the generated Swift header to the overlay directory. Adding it as a search path enables it to then be imported with #import “$(TargetName)-Swift.h”.

Does setting swiftSettings, cSettings, cxxSettings, and linkerSettings separately still pass the flags to the appropriate sub-target?

e.g.

let settings = [.unsafeFlags(["-Xfrontend", "-super-cool-swift-only-flag"], .when(configuration: .debug))]

...
let package = Package(
   ...
   .target(
      name: "MyMixedTarget",
     dependencies: [],
     swiftSettings: settings

Thanks

3 Likes
3

Thanks @etcwilde– answers below!

  1. CMake mixed targets

I double checked within my implementation, and I never touched any code that had anything to do with linking. But I recalled that I did add tests that build and integrate a dynamically linked mixed target and a statically linked one, respectively. I just built each one so I could inspect the build description file (debug.yaml) in the .build directory. I looked for the node responsible for linking– I'm thinking this would be the link phase you are asking about. I copied and pasted each node into a public gist so you can see what happens exactly to create the .a or .dylib. Let me know if this answers your question!

  1. C++ interop

At the time of writing this proposal, the C++ interop flags were still experimental, so I opted for supporting C++ with some, albeit considerable, limitations using the non-experimental path. In the time since, I did investigate using the interoperability flags on top of this proposal’s current implementation and saw some promising results. Fast-forward to now, and I think one goal of this review cycle should be to determine whether or not C++ support via the latest interoperability advancements is in-scope (as opposed to being added on in a follow-up proposal). I appreciate any feedback here.

  1. Specifying flags to clang build job vs Swift build job vs Link step

Just to recap from earlier in the proposal, a MixedTargetBuildDescription type is being introduced that consists of an underlying ClangTargetBuildDescription and SwiftTargetBuildDescription. The latter two types of which already exist. My intention in this section was to explain how the flags used in each build description need to be adjusted so that Swift sources are built with the needed knowledge of the Clang sources and vice versa. Let me know if this makes things less confusing– I’m happy to re-work this section as needed.

This is a great question. I haven't handled this in any special way and while I haven't ran into any issues during testing, I suspect there could be some issues here. I'll take a closer look to make sure language specific flags are passed to the appropriate sub-target. Thank you for bringing this up!

1 Like
4

I think it would make sense to support C++ as well, as it's now supported in Swift 5.9 . The implementation of the proposal needs to ensure that the .interoperabilityMode(Cxx) setting is respected by the Swift code in mixed-language target, and everything else should fall into place I think.

For this specific statement

Additionally, when present, each non-Objective-C/C header (e.g. a C++ header) is excluded from the module. This avoids build errors that arise when exposing, for example, a C++ header to Swift

I think mixed-language targets should not exclude C++ headers from the module, even when C++ interoperability is not enabled, as that could be an obstacle in the future that makes it more difficult for us to make C++ interoperability a feature that's on by default or doesn't not require a flag/setting to turn on. Users who don't use interoperability would need to then guard their C++ in headers in #ifdef cplusplus blocks. Also relying on an extension of the header is not necessarily 100% accurate , as people commonly place C++ code into .h files as well.

This module map defines an umbrella directory at the target’s path, exposing all headers for the target. When used to build the target’s Swift sources, the target’s Swift sources can use all types defined in the target’s headers, as opposed to only types declared in the target’s public headers.

It might also be good to figure out some way to allow users to exclude headers from the module that's brought into Swift, as some headers might not be modularizable, for instance .inc files that are then always textually included into other headers.

3 Likes
5

Thanks @Alex_L, I agree with your points! So, to sum up what C++ interop support could/should look like for this proposal then:

  1. Authors of mixed Swift/C++ targets, would need to explicitly pass the .interoperabilityMode(Cxx) Swift setting to opt in.
  2. For authors of mixed Swift/C++ targets that don't want to opt in, they're responsible for guarding their C++ headers appropriately. This would address one of the open questions I had within the implementation, where the package manager would instead need to decide based on file extension (which is not ideal for the reason you mention).
  3. Some header formats might not be modularizable, so authors should have a way to exclude them.

I'd like to better understand your last point (#3).

In the case of .inc files, is the issue that there is no way to export the interfaces defined within an .inc file to Swift, so therefore they need to be excluded from the module that's brought into Swift? Could this Why wouldn't #ifdef cplusplus blocks (#2) work in such cases? For what it's worth, my C++ knowledge is limited so I'd definitely like to better understand this.

1 Like
6

Will there be any changes to the PackagePlugin APIs?
SwiftSourceModuleTarget and ClangSourceModuleTarget

3 Likes
7

Hi @benrimmington, just to confirm, the question relates to enabling a package plugin to process a mixed language target? If so, it is something I had not considered, and will look into. Depending on what this entails, I think it may make sense to either include it in this proposal's scope, or pursue it in a follow-up to this proposal.

1 Like
8

The issue would be specific to Clang modularization, and what set of headers form a valid Clang module, not whether some header interface can be brought into Swift. For instance, some headers can redefine declarations that cause problem only when the set of headers is built as a Clang module, but not when headers are directly included into the compiled C/C++ translation unit.

However, upon a second thought, I think additional support for exclusion is probably not really necessary since your proposal still allows users to define a custom module map, which will allow people to write a custom module map that only covers headers of interest instead of excluding some.

1 Like
9

The commands in the gist look good to me.

Yeah, I agree with @Alex_L here. Now that C++ interop is out in 5.9, we should at least have a plan on what that integration looks like. I wonder if it would make sense for the compiler to eventually always emit both the C entry points and the C++ entry points wrapped in an #ifdef __cplusplu macro into the same header so that C and C++ sources can just pull in the same header and use the relevant pieces?

2 Likes
10

If you're talking about the generated header here, then that's what we already almost do. You still need a flag to enable interop, but once it's set we'll generate a unified header with C and C++ sections. If you don't enable interop we'll just be missing the C++ section and the related Swift native C function definitions that are needed by the C++ section. Eventually it should be on by default (for the header generation, not necessarily for C++ -> Swift import) though I agree.

3 Likes
11

Clang modularization when Swift/C++ interop mode is NOT enabled

So I've thought more about this, and have a follow-up question. Taking a step back to add some context before getting to my question... as things are currently designed, there are two module maps involved in the building of the target.

  1. The product module map exposing the Clang module's public API. SPM generates one if the package author does not include one in their public headers directory. In the proposal, I refer to this as the "product" module map as it's part of the target's build artifacts used by clients.
  2. The intermediates module map is a concept I'm introducing as part of this proposal. Its purpose is to expose all headers to the Swift implementation within the target. This enables developers to use types defined in non-public headers in their Swift implementation. This is supposed to be an improvement over mixed language frameworks, where the Swift implementation can only access the types defined in public headers. In the proposal, I refer to this as the "intermediates" module map as it's an intermediary build artifact.

The "template" for generating the intermediates module map looks like this:

module {ModuleName} {
  umbrella {PATH/TO/TARGET_SRC_ROOT}
}

I'm using an umbrella directory to just grab everything. As discussed, package authors should guard C++ headers with #ifdef __cplusplus, so such headers are no longer a problem that needs to be handled in the module map (this is where previously I was hackily adding exclude header statements).

Does this module map structure seem reasonable? I'm wondering if there is some obvious use case where a C++ developer includes a file type (maybe .inc?) that can't be guarded with #ifdef __cplusplus and therefore this module map includes non-moduralizable files.

Clang seems to know what files in the umbrella directory to process (e.g. it doesn't complain that it's finding .swift files).

12

Configuring build settings for a mixed language target

@etcwilde, I took a closer look, and wanted to raise an interesting point with regard to this.

Yes, flags are passed to the appropriate sub-target, but there are cases where language-specific flags are passed to the other language's sub-target. This looks intentional based on the existing implementation on main. One example is how cSettings are passed to the Swift sub-target as well as the Clang sub-target. I added some testing coverage that demonstrates this. You can see how the C setting was passed to the underlying Swift build command (in addition to the Clang command, of course).

Currently, Swift targets get their build settings from several settings configured by the package author (source). Likewise, for Clang targets (source).

So each language-specific sub-target may grab a setting from the other language sub-target's build setting. This is the point I wanted to raise as it might be considered unexpected behavior for the author of a mixed language target to see this in action.

If this is considered an issue, then we'd likely need to add to the Target public API to have something like swiftOnlySettings, cOnlySettings, etc.

I personally can't think of any use cases where this behavior would be a blocker, so I would lean towards just documenting the current behavior for now. Something to spell out that setting a cSetting doesn't necessarily mean it's only applied to building the C-Language source.

13

This is expected for the clang-importer so that Swift can import definitions in a way that is consistent with how C/C++ would see it.

"-Xcc", "-DHELLO_CLANG=1"

The -Xcc from the test says that the definition is being passed through to the clang-importer, so it shouldn't affect the Swift code itself.

So if that's how it's being passed around, then I think this makes sense. I would be surprised if Swift settings are passed to clang though.

1 Like
14

Aha, thanks– makes sense to me!

By the looks of it, they are not.

15

That certainly looks reasonable, as long as there's a way for the user to manually provide the intermediates module map for their sources instead of relying on the generated one.

16

Curious, is this coming from a specific use case you're thinking of, or is it more of a way to give authors an out when an edge case issue surfaces with the generated module map's format?

In the feature's current state, there isn't a way to pass an intermediates module map, and I'm not sure what one would look like. A lot of the difficulty in getting the intermediates module map to work in the first place was learning/using Clang's VFS Overlay system to lay things out– especially in cases when the package author provides a custom product module map.

I think what I'm effectively emulating with the intermediates module map is a private module map (Modules — Clang 18.0.0git documentation). Borrowing the naming, the DevX could be package authors would add a module.private.modulemap to the root of their target's sources directory...

On the note of Clang's private module maps, I learned about these after arriving at the current solution that relies on the Clang VFS overlay system. Since, I've used them in CocoaPods projects to achieve the same thing as what I achieve with the intermediates module map. The only downside is you usually have to import the private module like import MyModule_Private rather than the implicit importing that exists with the current SPM solution I've put together. I'm not sure they (as described in the Clang docs) could be used in the SPM context since SPM targets are not built as frameworks and I've never found a "private module map" specific flag. Just throwing this out there in case you or anyone else is more familiar with Clang's private module maps.

I'll see what I can do.

17

This is definitely more so for the edge cases. These tend to crop up here and there when trying to modularize existing libraries and their headers. Things get much worse when you add C++ into the mix.

Also, reading over your proposal again I'm a little unsure about exactly why you need the full intermediates module.modulemap for Objective-C sources specifically. Is it required specifically for building C and Objective-C sources with clang modules enabled? If you build the sources without Clang modules enabled, do you no longer need the full intermediates module.modulemap, and just need the unextended intermediates module map for importing the headers into Swift, right?

Another thing I noticed in the proposal:

Below is an example of a module map for a target that has an umbrella header in its public headers directory (include).

// module.modulemap

// This declaration is either copied from the custom module map or generated
// via the rules from SE-0038.
module MixedTarget {
    umbrella header "/Users/crusty/Developer/MixedTarget/Sources/MixedTarget/include/MixedTarget.h"
    export *
}
// This is added on by the package manager.
module MixedTarget.Swift {
    header "/Users/crusty/Developer/MixedTarget/.build/.../MixedTarget.build/MixedTarget-Swift.h"
    requires objc
}

Please do not add the requires objc to the Swift submodule. This would make it impossible to import without Objective-C interop enabled but with Clang modules enabled.

Another thing that stood out to me was the use of absolute paths, even for things like the generated header. I've experienced some bugs with using absolute paths for generated module maps in the module map for specifying headers before so I'm a little weary. I think perhaps for the umbrella directory it's fine, but could we map the generated header file to "MixedTarget-Swift.h" instead of using the absolute path in the intermediates directory for better reliability? IIUC, the overlay already maps to be located next to the intermediates module map, so using the header name directly should work right. I think in Xcode the generated module map uses relative path to the generated header. You can even remap the umbrella directory using a relative path too as a subdirectory in the intermediates module map, if you add an overlay entry for it.

1 Like
18

I'm also very worried about the design whereby you're building a Swift module that has access to one specific view of a Clang module but its clients have a very different view of the Clang module, since the intermediates and the public module map refer to the same Clang module but their modules are actually very different since intermediates includes all the headers. This is problematic for interoperability between C/ObjC/C++ and Swift, since we serialize the references into the intermediates Clang module when building the Swift module and as such we'll have issues when a Swift compile invocation in the client of MixedTarget is loading them from the rebuilt public Clang module. In my very quick test it seems we could have silent miscompiles that go undiagnosed as we start dropping things like members of Swift structures when they're Clang types that are not present in the public Clang module. For example: interopMissingMOduleRefMiscompile · GitHub, this test shows that Swift is thinking that it's working with an empty Swift struct and it's miscompiling its use when it's presented with an incorrect view of the Clang module that's missing headers.

I think this proposal should instead present a consistent view of the Clang module for both the mixed target and other targets that depend on the mixed target.

1 Like
19

Great point– I just experimented and it turns out you don't need the full intermediates module.modulemap at all (commit). Thinking about it now, it doesn't make sense why it'd be needed. (I'm verifying against a mixed lang. targets test suite which I do feel covers the common package configurations well.)

Done (commit). I'd like to understand this better though since I missed it, what would an SPM package config that would expose this? A pure C++ SPM target trying to import a pure Swift SPM target? This was something I took from how Swift targets generate their module map on main. Does this edge case affect importing pure Swift targets too, or would it be limited to importing a mixed language target?

Yep, I understand! I will take a look... (edit: probably after addressing the larger design issue below)

20

I understand, thanks for raising. The goal of exposing the non-public headers was to enable library developers to incrementally migrate their library's non-public types to Swift without adding them to the public headers.

Given the current approach's flaws, totally makes sense.

I know I already mentioned it, but could Clang's private module maps (docs) be an opt-in solution to the original problem I tried to solve? I'm thinking of a solution where, by default, there is a consistent view of the Clang module for both the mixed target and other targets that depend on the mixed target, but the target's author can pass a module.private.modulemap that can than present an internal, explicitly-importable view of the Clang module.