1

This is a follow-up to the previous pitch by @typesanitizer.

Cross-import overlays allow Swift to automatically import additional “overlay” modules based on the combination of imports in a particular source file. They allow one library or framework to seamlessly offer tailored APIs for interoperating with another, without imposing additional dependencies or code on clients who don’t need it.

This feature has now been implemented and is in master and 5.3 nightly compilers, hidden behind the -Xfrontend -enable-cross-import-overlays flag. The proposal has been rewritten and reflects the implementation in the nightlies.

Major differences from the previous proposal:

  • The new proposal is far more specific now that implementation experience has shown us all the places it interacts with other language features. (This is part of why the Detailed Design section is pretty long.)

  • Re-exports through Swift modules count for cross-importing; re-exports through Clang modules do not. (Because C and Objective-C re-export everything imported into headers, their modules tend to transitively import way more than you think they do.)

  • Several paths have changed to allow bare libraries (non-frameworks) and Clang modules to declare cross-import overlays. (Cross-import overlays declared by Clang modules still only affect Swift imports, not C or Objective-C imports.)

  • The declaration files are in YAML, not JSON, to allow comments in the files.

  • The terminology is a bit different.

17 Likes
2

I've missed the previous pitch, but now having a closer look at this, I'm in favor in general, even though the implementation details seem a bit convoluted. I haven't seen similar features implemented in other languages, but I also haven't stumbled upon this exact set of issues in other languages. Are "peer dependencies" in NPM similar in any way?

I personally would prefer JSON to YAML. Even though JSON doesn't allow comments, at least it doesn't have weird type inference issues (such as no and yes values parsed as a boolean, not a string). Nevertheless, I realise that the YAML type inference issues are unlikely in this particular case, and if the presence of comments is required, I have no better alternative to propose (even though I wish that configuration languages like Dhall got some wider adoption). And as this is already present in 5.3 nightlies, the YAML ship has probably sailed.

Another thing I'd love to see from proposals and pitches going forward is to clarify the impact on Foundation and other core libraries if such is present. For example, I'd like to know if Foundation and Dispatch extensions for Combine are going to be moved to overlay modules. These implicitly imported extensions are a major pain when dealing with libraries such as OpenCombine. @broadway_lamb do I understand correctly that moving those extensions to an overlay module according to this proposal would solve those problems?

3

I guess not, since

  • Moving a declaration from a declaring module to one of its cross-import overlays is ABI-breaking.

Probably yes. But that's unlikely to happen.

4

The proposal says

Moving a declaration from a declaring module to one of its cross-import overlays is ABI-breaking.

It seems eventual adoption in Apple SDKs would be non-retrospective. So those specific declaration name clashes wouldn't be resolved, only the future ones.

5

Having a second thought, I wonder if we could "move" a declaration this way without breaking the ABI:

  1. Apply @available(*, unavailable) to the declaration in the declaring module, but keep the implementation.
  2. Copy the declaration to the cross-import overlay.

Supposedly, the compiler would pick the definition in (2) because unavailability implies lowest overloading precedence. Meanwhile, since the declaration and its implementation is kept in the original module, it is still available in the ABI for earlier clients.

cc @beccadax

6

I have no experience with YAML, but I’m curious why it was chosen rather than, for example, using Swift itself like SPM package manifest files do.

• • •

Also, and this is rather tangential to the proposal at hand, but in order to make separate modules truly interoperate with each other, it would be necessary to be able to retroactively make one protocol refine another.

For example, the Numerics module declares a Real protocol, and the _Differentiation module declares a Differentiable protocol. A full-featured cross-import overlay would be able to make Real refine Differentiable.

In this example the refinement would be unconditional, predicated merely upon both modules being present, but in general it speaks to a motivation for conditional refinements.

1 Like
7

Since this file is being parsed by the frontend itself, using Swift for its contents would require not only kicking off the parser (maybe not a big deal) but also defining an entire module with the types that would be used to describe the overlay and then type-check the overlay against that and probably execute it as SPM does its manifest. That seems like overkill, and significantly more complex than just validating the YAML structure.

There's already precedent for YAML being used by things like Clang's VFS overlays, and for folks who prefer JSON, YAML is a superset of JSON so you can stick to JSON in your own overlays if you want.

2 Likes
8

If you look at the example swiftoverlay file:

---
version: 1
modules:
- name: _KarrKit_Combine

It is 4 lines. I would guess that in the vast majority of cases, these files would be less than 10 lines total. Moreover, we do not need additional configurability for the features which are part of the proposal. As much as I like other configuration languages, including Dhall, incurring an additional dependency for a small use case (not to mention, now we need to go teach people a new configuration language for writing < 10 lines of code, or have them use something like a yaml-to-dhall), I don't think it's the right tradeoff here.

We also have precedent for using YAML for apinotes today. If there's a conversation about a new configuration format, we should consider the situation holistically instead of considering it independently when we introduce a new piece of configuration.

3 Likes
9

I really like how this proposal turned out!

Is this intended to go through evolution before @_exported? If not, would accepting this proposal require @_exported be eventually accepted without changes to avoid a source compatibility break? It's a little unclear to me how the experimental import features (@_private, @_implementationOnly, @_exported, and this proposal) are intended to all work together once they "graduate" to officially supported features.

5 Likes
10

I don't think this is strictly tangential, so perhaps we should discuss this a little bit. Having something like "retroactive refinement" as you suggest creates a problem. Let's extend your example to have the following modules:

  1. Numerics declaring a Real protocol.
  2. _Differentiation declaring a Differentiable protocol.
  3. A cross-import overlay _Numerics__Differentiation retroactively making Real refine Differentiable.
  4. A module M_N importing Numerics and having a type X : Real.
  5. A module M_ND importing Numerics and _Differentiation (hence obtaining the overlay) which tries to use X. Now, based on the retroactive refinement, one would expect that X : _Differentiation too. But that was not implemented in M_N... so where should the witnesses for X : _Differentiation come from?

Maybe the solution to this problem is that the retroactive refinement should provide default implementations for the protocols that are retroactively being refined. It's not clear to me if that would work for the different use cases where people would want retroactive refinement... maybe it works, maybe it doesn't. It also introduces a "you can only do this in an overlay" whereas the overlays in this proposal are ordinary modules with no special privileges in terms of what code you can write in the overlay.

In any case, the point I'm trying to illustrate here is that making something like that work is a bigger change to how the language works, and more complicated than it might seem at first glance.

In contrast, this pitch is focused on making minimal changes to the language while hitting what we think is a good point in the design space which covers many use-cases. We certainly have room to develop the feature with more evolution proposals down the line. :slightly_smiling_face:

1 Like
11

Format

This hasn’t been reviewed by evolution, so nothing has sailed yet. Having said that…

I’m sure you thought about this, but the issue here is simply that Dhall isn’t in LLVM’s compiler-building toolkit; YAML is, because the API notes and TBD (text-based dylib) files that already ship in frameworks are written in YAML. (As well as some ephemeral inputs, like the VFS files that Tony mentioned, but if we didn’t have YAML shipping in SDKs already, I would have thought twice.) It doesn’t make much sense to support a third general-purpose text serialization format (besides JSON and YAML) in the compiler.

Just too much effort for too little gain. To be honest, these were serious contenders:

  • Plain text list of newline-separated module names
  • Put all three module names into the path (like KarrKit.swiftcrossimport/Combine.swiftbystander/_KarrKit_Combine.swiftoverlay) and don’t read the files’ contents at all.

But having a version number and the ability to specify fields that are currently ignored seemed useful enough to justify the tiny engineering cost of using a structured format the compiler already understood.

Writing these files in Swift would have turned a day’s work into a month or two and also made loading slower. There was just no way to justify that.

(I also could have done is something like the modulemap format: use the Swift tokenizer but write a custom parser for the format. But again, this would have been a lot more work than using YAML for basically no gain.)

ABI

ABI compatibility issues would probably prevent us from fixing the Dispatch/Combine extension problem with cross-import overlays, but I think it would have been a great tool for layering issues like that one if we’d had it at the time. As the saying goes, the best time to add a feature plant a tree is thirty years ago; the second-best time is now.

I haven’t tried it, but at the very least, you wouldn’t be able to backwards-deploy calls compiled with the new SDK to older OSes, because they would be calling the new version instead of the old one.

This would be nice, but the fact that the set of protocols a protocol refines is known is baked into the ABI and I don’t think it can be changed at this point. (Varun talks about some of the reasons for this; I won’t duplicate his discussion.)

Other

It is, yes.

@_exported is in a weird spot: It’s not officially supported and we don’t want people over-using it, but the implementation is pretty good by now, and it’s so widely used and underpins so many important things (like traditional overlays) that removing it would be a de facto source compatibility break. It makes sense to support @_exported officially, but we’d have to decide how to design it to avoid misuse (and whether we want to reform import visibility more generally at the same time). This proposal just isn’t the place to make those decisions, and we don’t think it makes sense to hold up this proposal until those decisions are made.

It’s a bit awkward, but I think it’s the right decision. If @_exported gets an official spelling in the future, we can correct this proposal’s spelling so that it makes sense to future readers.

3 Likes
12

I’m very much in favour of the idea, however:

Add SwiftPM support to this proposal

SwiftPM support is a great idea and we would love to see it, but isn’t this proposal long enough already?

Strong no, can we please not create more language dialects? Without SwiftPM support I’m a strong -1 on this proposal. CC @tomerd / @lukasa

7 Likes
13

My opinion is not quite as strong as @johannesweiss given that this feature also relies on the unsupported @_exported import functionality, but I think this pitch should not lightly dismiss Swift Package Manager support. It is important for pitch authors to acknowledge that without SwiftPM support they are building functionality that the blessed language package manager community cannot use. These features essentially privilege Xcode and Apple platforms, as well as developers targetting those platforms, over the wider ecosystem.

My personal view is that this is out of sync with the stated goals for the road to Swift 6, specifically the goal to cultivate a rich open-source ecosystem, as well as its focus on broader platform support. This is mitigated somewhat by the fact that, due to its reliance on @_exported import, the feature is not stable for uses outside of the Apple SDKs anyway, but nonetheless, I would like to see this pitch either tackle the SwiftPM use case or clearly state that supporting SwiftPM is a non-goal at this time, with an explicit rationale for why, and what the plan is for addressing that limitation (if there is such a plan).

More broadly, it may be worth having the core team discuss whether it is acceptable to ship non-preview features that do not support Swift Package Manager uses.

15 Likes
14

For example, some teams write in a more imperative style; others use RxSwift; still others use Combine. Some backend teams use Vapor; others use Kitura; others use raw NIO. And the list goes on.

But this is a more general problem than just testing. It’s why ReactiveSwift, ReactiveCocoa, and ReactiveMapKit are three separate things. It’s why you need to import GRDBCombine separately from import GRDB . It’s why SwiftyJSON has a separate Alamofire-SwiftyJSON module. It’s why you have to think at all about half of the projects published under the RxSwiftCommunity organization.

All of the frameworks mentioned in the motivation section (except Combine) are SwiftPM packages. So how does this proposal solve the problem that motivated it without SPM support?

4 Likes
15

Sorry, that was probably too flippant. In more detail:

I think SwiftPM ought to support cross-import overlays. Off the top of my head, that probably means:

  • Allowing a target to say that it is a cross-import overlay, and specify the declaring and bystanding modules.
  • Given a target's declared dependencies, search among all targets (or products?) in the package graph for cross-import overlays it might load, and include them in the build.
  • Whenever a cross-import overlay target is in the build, generate the declaration files (i.e. the swiftoverlay files) when building its declaring module.

Evolution proposals usually only include language changes or package manager changes, however—not both—and this proposal is substantial enough that I don't think it makes sense to change our practice for it.

In other words, I think SwiftPM should support cross-import overlays; I just think that it's a separate proposal.

2 Likes
16

As somebody who is working with complex dependency graphs on daily basis, this proposal would make our setup so much simpler. It's a step in solving a real life problem that's definitely worth solving.

The proposal by itself, without SPM support, does not solve the problems from the Motivation section though. I think that that should be clearly acknowledged.

Regardless, a big +1 from me on this one.

2 Likes
17

Well, most of the time, they just work with SwiftPM though, right?

Totally fine by me, IF they get accepted only if the respective other gets accepted too. So I'm happy with two proposals that cross-depend on each other.

2 Likes
18

To function correctly, a cross-import overlay module should use an @_exported import of the declaring module and a regular import of the bystanding module:

// _KarrKit_Combine cross-import overlay

@_exported import KarrKit
import Combine

// Add new public APIs here

To hide the cross-import overlay from autocompletion, its module name should begin with an _ . We recommend using _<declaring module>_<bystanding module> as the overlay name, but this is just a convention—the compiler doesn’t ascribe any special meaning to this name.

Would it be better to create a properly supported compiler flag or some such that synthesizes these conventions when compiling a module for cross‐import? Then the use of @_exported would be a hidden implementation detail, and that would leave more room for @_exported to adapt as it approaches completion as a separate feature.

Dealing with them in two phases seems fine, as long as the first contains what it needs to make the second possible.

Points 1 and 3 seem straightforward, but I think point 2 needs at least a rough sketch.

Right now SwiftPM’s model is that things should only be fetched and built if they are actually needed, and so everything is explicit.

Under the current model, if you have a package with two products, MyModule and _MyModule_GiantExternalModule, then a client package only asking for MyModule doesn’t need _MyModule_GiantExternalModule, and never bothers resolving the transitive dependency on GiantExternalModule. Depending on MyModule could build near instantly, but depending on _MyModule_GiantExternalModule could take hours.

I can think of several different ways of trying to translate that to a world with cross‐import overlays:

  • Cross‐import overlays only affect the import statement; SwiftPM manifests still declare them like normal products. This would still be a significant improvement, because you have only one manifest, but many many source files, and each of those would still become more succinct. If this is what we’re aiming for, then we need to choose a naming convention that recognizes names will have to be referenced directly. In that case, I don’t think leading underscores are appropriate.
  • Cross‐import overlays are as implicit in the manifest as they are in source code. This is probably a better philosophical match to the intentions of the base feature. But how would that work?
    • Is the overlay always part of (or attached to) the base product? Then the monster build time of GiantExternalModule becomes an inherent part of the base product. Developers are likely to continue creating separate modules instead of overlays to keep the dependency trees separate.
    • Is the overlay only fetched if the import statements align such that it is determined to be needed? Then the base feature needs to be cleanly split into two phases, each separately accessible from the compiler’s interface:
      1. Pre‐parse all the module sources to determine and report which overlay combinations to look for. (So that SwiftPM or another build system can know what it needs to fetch.)
      2. Resolve the import during compilation after the required dependency build products are available.

There may be some other way forward that I haven’t thought of, but of those I brainstormed, 2 would have a significant effect on the design of the first phase (leading underscore, two‐phase approach) and the third would cause many package authors to avoid the feature (build‐time regression).

So I think we need to consider SwiftPM enough to have a rough idea of the way forward, and a reasonable certainty that it is implementable. That is the only way we can get this first phase right.

However, I do think this pitch has merit on its own, even if in the end we were to decide SwiftPM support will never happen and that the feature will only be available to platform vendors for their system modules.

2 Likes
19

My, perhaps naive, expectation would be that if the client module depends on MyModule and GiantExternalModule in its manifest, ir would automatically depend on _MyModule_GiantExternalModule, with any goop required to make this work living in MyModule’s manifest. It’s not clear to me why parsing imports would be necessary.

(Parsing imports could avoid the overlay dependency if MyModule and GiantExternalModule are never referenced in the same file, but avoiding compiling the overlay in that edge case doesn’t seem like it would be worth extra complexity)

2 Likes
20

I didn’t think of that intermediate option, because it wouldn’t work for combinations with non‐package modules (e.g. Something + SwiftUI). But since such combinations don’t actually add to the fetch or build process, it wouldn’t really matter. SwiftPM could ignore them and the compiler would handle it on its own from the imports.

SwiftPM could look up any overlays for dependency combinations needed by the same target, and that would be sufficient for 99% of use cases. (The remaining 1% that don’t use the imports in the same file could split their own module in two and stitch them together with @_exported to separate them from SwiftPM’s perspective so it wouldn’t fetch what isn’t needed.)

On the other hand, that reminds me that @_exported (or whatever its final spelling ends up being) can forward a transitive dependency upward, but that forwarding isn’t (currently) described in the manifest anywhere.

For each source file, the compiler computes the set of modules that are visible in that file, including via @_exported import s from Swift modules. Then it looks for combinations of these modules that have cross-import overlays. If it finds any, it implicitly imports those cross-import overlays.

How would that work? If the bottom module is “internal”, the middle module @_exports it as part of product, and the top module imports the middle one, then the top’s manifest has no means of declaring any relation to the bottom module, so its overlays wouldn’t be fetched for the compiler to see.

I guess that could be solved by adding some means of declaring the @_exported import in the manifest. It wouldn’t be source‐breaking, because old @_exported imports would still work without it, they just wouldn’t have access to overlays. And the overlays wouldn’t have previously accessible anyway.

Does that sound right, or have I missed something?