[Pitch] Adding support for targets with mixed language sources

Evolution Pitches
1

Hi everyone –– I'd like to present the following pitch for adding support for targets with mixed language sources (e.g. Swift + Objective-C). The corresponding pre-pitch can be found here.

Package Manager Mixed Language Target Support

Implementation: apple/swift-package-manager#5919

Introduction

This is a proposal for adding package manager support for targets containing both Swift and C based language sources (henceforth, referred to as mixed language sources). Currently, a target’s source can be either Swift or a C based language (SE-0038), but not both.

Swift Evolution Review Thread

Motivation

Packages may need to contain mixed language sources for both legacy or technical reasons. For developers building or maintaining packages with mixed languages (e.g. Swift and Objective-C), there are two workarounds for doing so with Swift Package Manager, but they have drawbacks that degrade the developer experience, and sometimes are not even an option:

  • Distribute binary frameworks via binary targets. Drawbacks include that the package will be less portable as it can only support platforms that the binaries support, binary dependencies are only available on Apple platforms, customers cannot view or easily debug the source in their project workspace, and requires tooling to generate the binaries for release.

  • Separate a target’s implementation into sub-targets based on language type, adding dependencies where necessary. For example, a target Foo may have Swift-only sources that can call into an underlying target FooObjc that contains Clang-only sources. Drawbacks include needing to depend on the public API surfaces between the targets, increasing the complexity of the package’s manifest and organization for both maintainers and clients, and preventing package developers from incrementally migrating internal implementation from one language to another (e.g. Objective-C to Swift) since there is still a separation across targets based on language.

Package manager support for mixed language targets would address both of the above drawbacks by enabling developers to mix sources of supported languages within a single target without complicating their package’s structure or developer experience.

Proposed solution

This solution allows the package manager to determine if a target contains mixed language sources and build it as a single module. It's done automatically and doesn't require changes to the package manager's public API.

At a high level, it splits the package creation process into two parts based on the language of the sources. The Swift sources are built by the Swift compiler and the C Language sources are built by the Clang compiler. To achieve interoperability between the two halves of the package, a few things have to happen:

  1. The Swift compiler is made aware of the Clang half of the package when building the Swift sources into a swiftmodule.
  2. The generated interoperability header emitted by the Swift compiler is added as a submodule to the Clang half of the package’s generated module map.
  3. The Clang half of the package is built with knowledge of the generated interoperability header.

The following example layout defines a package containing mixed language sources.

// Manifest
MixedPackage/Package.swift

// Clang sources
// - Public headers
MixedPackage/Sources/MixedPackage/include/MixedPackage.h
MixedPackage/Sources/MixedPackage/include/Sith.h
MixedPackage/Sources/MixedPackage/include/droid_debug.h

// - Implementations and internal headers
MixedPackage/Sources/MixedPackage/Sith.m
MixedPackage/Sources/MixedPackage/droid_debug.c
MixedPackage/Sources/MixedPackage/SithRegistry.h
MixedPackage/Sources/MixedPackage/SithRegistry.m

// Swift sources
MixedPackage/Sources/MixedPackage/Jedi.swift
MixedPackage/Sources/MixedPackage/Lightsaber.swift

// Resources
MixedPackage/Sources/MixedPackage/hello_there.txt

// Tests
MixedPackage/Tests/MixedPackageTests/JediTests.swift
MixedPackage/Tests/MixedPackageTests/SithTests.m

The proposed solution would enable the above package to have the following capabilities:

  1. Export the public API of the mixed language sources as a single module for use by clients of the package.
  2. Use any Objective-C compatible API exposed by the package’s Swift sources within the package’s Objective-C sources.
  3. Use any public API exposed by the package’s Objective-C or C sources within the package’s Swift sources.
  4. Use internal C based language sources within the Clang half of the module. Likewise for internal Swift types within the Swift half of the package.
  5. Access target resources from a Swift or Objective-C context.

Requirements

Initial support for targets containing mixed language sources will have the following requirements:

  1. The target must be either a library or test target. Support for other types of targets is deferred until the use cases become clear.
  2. The target must be built on a Mac. This is because the Swift compiler-generated Objective-C compatibility header is only generated on macOS.
  3. Including a custom module map is not supported. This is to reduce the possibility of an invalid module map since a custom module map will need to be modified by the package manager to add a submodule to expose the generated Objective-C compatibility header. Support for this is deferred until the use cases become clear.

Detailed design

Up until this proposal, when a package was loading, each target was represented programmatically as either a SwiftTarget or ClangTarget. Which of these types to use was informed by the sources found in the target. For targets with mixed language sources, an error was thrown and surfaced to the client. During the build process, each of those types mapped to another type (SwiftTargetBuildDescription or ClangTargetBuildDescription) that described how the target should be built.

This proposal adds two new types, MixedTarget and MixedTargetDescription, that represent targets with mixed language sources during the package loading and building phases, respectively.

While an implementation detail, it’s worth noting that in this approach, a MixedTarget is a wrapper type around an underlying SwiftTarget and ClangTarget. Initializing a MixedTarget will internally initialize a SwiftTarget from the given Swift sources and a ClangTarget from the given Clang sources. This extends to the MixedTargetDescription type in that it wraps a SwiftTargetDescription and ClangTargetDescription and configures them accordingly to successfully build.

To compile both sources, the Clang Virtual File System overlay system is used to pass necessary metadata and file references to the Clang compiler. Specifically, a temporary module map is substituted when compiling the Swift part of the module because the real module map includes a reference to the Swift generated header that does not exist yet. This approach was, in part, informed by this Swift forums discussion.

The proposed solution adds the following behavior:

  1. During the package loading phase, a target with mixed language sources will be represented by a MixedTarget.

  2. During the package building phase, the MixedTarget will be used to create a MixedTargetBuildDescription to inform the build process.

  3. Initializing a MixedTargetBuildDescription will do a few things, both directly and indirectly.

  4. Generate a module map that exposes the public Clang sources and includes a submodule for the interoperability header that will generate during the build.

  5. Generate a Clang VFS overlay of the module’s public headers and module map.

  6. Generate a Clang VFS overlay of a modified module map that excludes the submodule with the generated interoperability header.

  7. Pass the --import-underlying-module flag to the underlying SwiftTargetDescription’s build flags.

  8. Pass the two VFS overlays to the SwiftTargetDescription’s build flags. The Swift compiler will pass these to the underlying Clang compiler.

  9. Pass the directory of the generated interoperability header to the ClangTargetDescription’s build flags.

  10. When building a MixedTarget as a dependency of a parent target, pass the module map and/or public header directory to the parent target's build flags. This follows the exact behavior of when building a ClangTarget as a dependency of a parent target.

  11. Construct the llbuild manifest (found at $(PackageName)/.debug/debug.yaml) to include the Swift compiler command to built the Swift half of the target and the Clang compiler command to build the Clang half of the target. The Clang command is always executed after the Swift compile command as it depends on the interoperability header generated by the Swift compile command as an input.

One benefit of this design is that it offers a natural path to making all targets mixed source targets by default. This would greatly simplify logic in the package loading and building phases. With this approach, the implementation from ClangTarget, SwiftTarget, ClangTargetBuildDescription and SwiftTargetBuildDescription can be bubbled up to the mixed target types accordingly.

Security

This has no impact on security, safety, or privacy.

Impact on existing packages

This proposal will not affect the behavior of existing packages. In the proposed solution, the code path to build a mixed language package is separate from the existing code paths to build packages with Swift sources and C Language sources, respectively.

Alternatives considered

Provide custom implementations for MixedTarget and MixedTargetBuildDescription

As explained in the Detailed Design section, these two types effectively wrap the Swift and Clang parts necessary to define or build the target. One alternative approach was to provide custom implementations that did not heavily rely on code reuse of existing types. The deciding drawback of this approach was that it would have resulted in a lot of duplicated code.

Future Directions

Some of these are informed by current constraints from the above Requirements section.

  • Extend mixed language target support to other types of targets (e.g. executables).
  • Extend support for mixed language targets with custom module maps.
  • Expand the level of support when building on non-macOS machines.
  • Extend this solution so that all targets are mixed language targets by default. This would simplify the current implementation of the package manager.

Open questions for this Pitch

Please ask and discuss questions about any part of this pitch. I have identified the following questions while working on the solution that I’d like to resolve.

  1. The solution’s implementation at this point does omit support for custom module maps. While there is a path to support them, I’d like to hear from the community about whether that should be pursued. The reason I have flagged this is because the module map needs to include the generated Swift header, but that is only known at build time. Supporting custom module maps would mean taking the custom module map and programmatically adding to it during the build process. This would not be very intuitive if issues arise. What do folks think about supporting custom module maps for mixed language targets? What do folks currently like about defining custom module maps for targets with Clang-only resources?
  2. Supporting types of targets other than library or test targets is not straightforward, and will require some more investigation. Do folks have strong feelings about supporting other types of targets as well?
  3. Currently, the package manager will only generate the Swift interop header on macOS. As such, trying to build a mixed target on Linux, for example, would throw an error. There are cases though where this constraint may not limit a mixed language target. For example, building a Swift + C source target on Linux. The generated Swift interop header isn’t relevant here, so it may be able to build successfully. I wanted to flag this so folks in favor of such behavior could provide use cases to justify looking into this further.

Thank you for reading and discussing this pitch!
– Nick from the Firebase team

26 Likes
2

As regards to improving language support for SwiftPM, I wonder if the new implementation will address [SR-14728] SPM/Driver on Windows fails to link C-based targets · Issue #4415 · apple/swift-package-manager · GitHub, and guarantee that Swift runtime won’t be linked for a target with no Swift source?

To expand some bit on it, I personally really like the idea of using SwiftPM for managing C/C++/ObjC projects, but there are many problems in practice, mostly caused by SwiftPM’s prioritizing Swift and, thus, breaking non-Swift targets. Besides automatically linking Swift runtime, it seems only Swift targets can consume APIs exposed by other targets, and non-Swift targets are just independent “producers”. I wonder if the SwiftPM team has noticed such problems, or is such behavior exactly intended?

3

Hi @stevapple, I took a look and I don't think this pitch's current implementation will address that issue because a "mixed" target is created only when Swift and Clang sources are detected. For targets with no Swift source, this pitch's implementation will have no effect.

As for the second part, I'll let the SwiftPM team jump in.

4

Will this proposal also allow mixed Swift and C++ targets in SwiftPM?

5

This proposal should support targets that contain both Swift and C++. Communicating between Swift and C++ source, if desired, will require using an Objective-C++ layer within the target.

2 Likes
6

Thanks for your pitch, Nick! I think overall the approach looks very solid to me.

Regarding the open questions, 2+3 seem reasonable to keep as-is to me, but supporting custom module maps seems important, just from the perspective that if someone has an existing C target, they should be able to "just" add Swift files to it. Could we let package authors add the generated Swift header to their custom module maps manually? Or is that not feasible because it requires knowing the absolute path to it?

1 Like
7

Thanks @NeoNacho, I agree that supporting custom module maps may be important.

We could let package authors add it manually because the path to the generated Swift header is deterministic. The path should always be /path/to/$(PackageName)/.build/x86_64-apple-macosx/debug/$(PackageName).build/$(PackageName)-Swift.h so authors could fill in the blanks and add it to their module map. A relative path to the header from where their module map is located should work as well.

I can think of two concerns with regard to this approach:

  1. This may not scale for when building with Xcode (I'm assuming this header lives in a different place than the SPM .build directory)
  2. Putting SPM-specific code in their module map may be problematic when using their module map in a non-SPM context.

Both of these would be addressed if we tweak SPM (and, later, Xcode's build tool) to modify a custom module map to include that header. Like I mentioned in my original post, it's a bit sneaky but could be acceptable with good documentation and quality error messages for edge cases.

8

OK, I think requiring authors to know the absolute path doesn't scale very well.

Two possible ideas for a less sneaky approach:

  • copying the custom module map to wherever the generated module maps and headers get placed, so that a relative path would work
  • offer some kind of variable/token authors can use in their module map that we predictably replace instead of adding new statements to the custom module map, e.g. if the authors writes something like header "$(GENERATED_HEADER_PATH)/MyModule-Swift.h", we would replace GENERATED_HEADER_PATH accordingly.
1 Like
9

One caveat to idea #1 is that the custom module map may contain a relative path to begin with. So copying it would then break that relative path. I like the idea of copying the module map, though. I think we would then need to detect any relative paths in the module map copy and add a path prefix to preserve each relative path. I can't think of a way around this, especially because "files listed in module maps are not found through include paths" (from the Clang modules docs).

I think idea #2 is a good option as well. Though, it would mean putting SPM-specific code in the module map. Therefore, package authors may have to manage more than one module map if, say, they distribute their project as a SwiftPM package and a CocoaPod. Both of the package managers support excluding files so it'd be an inconvenience more than anything.

Additionally, I'll look into what CocoaPods does when you have a mixed pod and specify a custom module map using the spec.module_map field. I'm curious how they approached it.

10

What about using -vfsoverlay to specify the location of a virtual module root that has the module map, user headers, and Swift-generated header all mapped into it, without having to move or copy the actual files around?

IIRC Xcode uses some form of this approach when doing the multi-phase compilation required for mixed Swift/Obj-C modules.

1 Like
11

Thanks @allevato. So the pitch's implementation currently uses -vfsoverlay to specify just that (code snippet). This, in addition to the --import-underlying-module, is passed to the Swift compile command and I believe used by the Clang compiler when building the underlying module.

The reason the generated Swift header should be in the module map is for when a Clang-source target wants to use a public Swift API from a Mixed-source target.

Are you suggesting there is a way to pass an overlay file using -vfsoverlay to avoid having to specify the generated Swift header in the module map? I just finished some experimenting and did not find a way to do so.

12

Ah, no, you would still need to modify the module map, but the -vfsoverlay would help in letting you put it in the regular generated files location (as you've done, thanks for the link!).

I haven't scoured the implementation fully yet so I'm not sure if this is along the lines if what you're thinking, but do we care whether the Swift generated header is in its own submodule? If not, this makes it a lot easier, because you don't have to do any awkward parsing of the module map file or ask the user to write anything with placeholders; you can just concatenate the module map with a new submodule declaration. For example, if the user provides this:

module MyCoolModule {
  header "MyCoolClass.h"
  header "MyOtherCoolClass.h"
  export *
}

Then SPM could copy it and append the following to it:

// ...existing module.modulemap contents...

module MyCoolModule.Swift {
  header "MyCoolModule-Swift.h"  // generated
}

and then you use the VFS to put the modified module map and generated header in the same virtual location as the user's headers.

Edit: Naturally this means the user couldn't introduce a submodule of their own named "Swift", but it seems like a reasonable restriction to have in place and it could be easily documented.

Edit 2: I just tested this with Xcode by adding a custom module map to a framework project and it looks like that's exactly what it does as well.

1 Like
13

Yep, I understand. That is inline with what the implementation is doing now –– adding a generated Swift header in a submodule (code snippet)

So this is good and works well. Now, when the package author has a custom module map, I wasn't sure if we should do this because of the edge case where they may have a Swift submodule.

While typing this, I see your edit:

So you think it's reasonable to have this restriction on custom module maps and document that custom module maps will be edited to include the Swift submodule?

1 Like
14

I personally think that's totally reasonable (especially since I just discovered that it's what Xcode is doing too). It's a small restriction that makes the user experience completely transparent instead of needing them to modify their module map with a magic placeholder string.

If we're concerned that someone would collide with this, I think you could offer an optional argument on the target that lets them change the name of the Swift submodule. But that seems like a rare enough case (unless you've seen occurrences of it in the wild) that it may not even be worth the overhead.

2 Likes
15

Just to clarify, Xcode modifies the custom module map directly (1) or does it copy it, modify it, and pass around an overlay so the original module map remains unchanged (2)?

And thank you very much for the feedback and experimenting!

16

If you provide a custom module map in the "Packaging > Module Map File" build setting, then the original file is left untouched. Xcode copies it twice into its intermediate build outputs and modifies the copies, and puts those copies into the right place via the VFS:

  • A copy named module.modulemap that appends the Swift submodule with the header declaration for the generated header
  • A copy named unextended-module.modulemap that appends the __Swift submodule with the exclude header declaration for the generated header
1 Like
17

Understood! That's awesome because I assume that substituting a modified overlay doesn't break any relative paths in the custom module map because it's ~virtually~ located at that path.

Edit: I'll adjust the implementation accordingly and see what happens.

1 Like
18

Update –– supporting custom module maps was indeed feasible with the approach discussed above. There were a few bumps along the way, but I ultimately arrived at the desired behavior (see commits 669225c and 44377cf).

2 Likes
19

I wanted to share another update–– I've added two new capabilities to this pitch's implementation.

  1. Support sharing of test utilities within a single mixed test target
    • This pitch allows for creating test targets containing both Swift and Objective-C test files. Recently added is the ability to use an Objective-C test utility (e.g. MyTestHelper.h) in a Swift test file, and vice versa–– use a Swift test utility (e.g. MyTestHelper.swift in an Objective-C test file).
  2. Expose all headers (including non-public headers) to the Swift-half of the mixed target
    • Up until now, the Swift half of a mixed target only had access to the public headers of the Clang half of the target. This matches the behavior of building a mixed language framework. One shortcoming of this design is that the Swift implementation does not have access to internal/private headers. For example, this makes migrating a library from Objective-C → Swift more difficult since private headers must be made public to expose them to the Swift half of the module–– which is not ideal. To work around this, I use one module map that includes all headers (via an umbrella directory) to build the mixed target, and use another one for clients of the target to use (via -fmodule-map-file=) that only exposes the mixed target's public API surface. Just like with supporting custom module maps above, the VFS overlay system came in handy to help make this work. I'm continuing to test this, but so far it is working great.

Relevant commit for both features.

5 Likes
20

Hi everyone, just wanted to update this thread that the formal proposal PR is up!

It features a revised Detailed Design section that sums up the current state and thought behind the implementation (apple/swift-package-manager/#5919).

Thanks everyone for the feedback on this Pitch. :slightly_smiling_face:

3 Likes