Cross-Import Overlays

Evolution Pitches
1

Cross-Import Overlays

The problem

[Note: I've added new information to this section with some additional benefits of this approach after reading the comments. These points are prefixed with a ★.]

Say we have a pair of frameworks NumericKit and FormatKit where NumericKit covers numerical simulations and FormatKit covers serialization and deserialization for a bunch of data formats. Ideally, a downstream project that depends on both of these frameworks would "automatically" get cross-cutting functionality. In this example, it could be something like serializing complex data using the HDF5 format (say).

If one wants the automatic behavior today, the cross-cutting functionality has to either live in NumericKit (in which case it depends on FormatKit) or FormatKit (in which case it depends on NumericKit). This is a dissatisfying solution (★ especially when both NumericKit and FormatKit are not system frameworks) because it means that

  1. clients potentially incur dependencies which are unused: (★) this increases the amount of code that might need to be audited, especially if it is relying on a C/ObjC core for functionality
  2. (★) clients incur additional compile time + link time overhead:
    a. It reduces build parallelism because swiftmodules are built in dependency order.
    b. It increases the amount of code one needs to compile and link.

The other option to have a separate framework, say NumericKitFormatKitAdditions for the cross-cutting functionality, which solves the problems 1, 2a and 2b mentioned above. ★ Such cross-cutting functionality not only extensions for third party types but also conformances. We would generally like people to avoid writing retroactive conformances with increasing adoption of library evolution; such an Additions framework would provide a natural home for retroactive conformances without issues.

However, this solution requires downstream users to write import NumericKitFormatKitAdditions which is inconvenient and a bit ugly. If there is just one such module, it is probably fine. However, if a framework's functionality overlaps with many other frameworks, this can lead to a displeasing wall of imports:

import NumericKitFormatKitAdditions
import NumericKitPlotKitAdditions
import PlotKitFormatKitAdditions
import NumericKitTestKitAdditions
import FormatKitTestKitAdditions
import PlotKitTestKitAdditions

Wouldn't it be nice if we could mark these "Additions" modules in a way such that if you wrote

import NumericKit
import FormatKit
import PlotKit
import TestKit

(which you are probably writing anyways) you automatically get the cross-cutting functionality as well?

The proposed solution

The typical case (N = 2)

We propose a new language feature 'Cross-Import Overlays' that allows framework authors to provide cross-cutting functionality in a modular fashion without having to burden users with additional import clutter.

Consider the example shown earlier. Say if NumericKit decides to expose an overlay for FormatKit, NumericKit can add a new JSON file FormatKit.json under an Overlays subdirectory in its swiftmodule.

NumericKit.framework/Modules/NumericKit.swiftmodule/Overlays/FormatKit.json

The contents of this JSON file will tell the compiler what the name of the overlay module is.

{
  "version": 1,
  "modules": [
    {"name": "_NumericKitFormatKitAdditions"}
  ]
}

Here, version represents the scheme version for the JSON file and _NumericKitFormatKitAdditions is the name of the cross-import overlay. As a convention, we recommend that the name consist of an underscore, followed by the name of the primary underlying module (here NumericKit), followed by the name(s) of the secondary underlying module(s) (there’s just one here: FormatKit) followed by Additions.

(Aside: The exact naming scheme is up for discussion, this need not be the final one. One benefit of it being a convention, instead of say a new symbol like +, is that we don't need to change the grammar, parser and potentially other parts of the compiler and downstream clients.)

Once the compiler sees this module name, it can try to find a module for NumericKitFormatKitAdditions using the usual search paths.

The expected behavior would be that when the compiler sees both NumericKit and FormatKit, it will try to find _NumericKitFormatKitAdditions, and if successful, the resulting code behaves as if import _NumericKitFormatKitAdditions had been explicitly written in the source.

The general case (N >= 2)

Say we want to add a cross-import overlay which has three underlying modules: NumericKit, FormatKit and TestKit. In such a case, we can add the following two JSON files:

# same as before
NumericKit.framework/Modules/NumericKit.swiftmodule/Overlays/FormatKit.json # contains 
# new overlay
_NumericKitFormatKitAdditions.framework/Modules/_NumericKitFormatKitAdditions.swiftmodule/Overlays/TestKit.json

[Note: the second path was initially using NumericKit.framework, which was incorrect, as pointed out below. Thanks @Xi_Ge and @allevato.]

Once the compiler sees this directory structure, and all three of NumericKit, FormatKit and TestKit are imported, it will automatically import the two other modules as well

import _NumericKitFormatKitAdditions        // via FormatKit.json
import _NumericKitFormatKitTestKitAdditions // via TestKit.json

Impact on name lookup

We recommend the following shadowing behavior:

  1. A cross-import overlay’s declarations shadow those in the primary underlying module.
  2. A cross-import overlay’s declarations do not shadow those in the secondary underlying module(s)

The second rule has a chance of creating ambiguity: what if there is a name collision between a declaration in a cross-import overlay and a declaration in one of its secondary underlying modules?

There is an existing pitch Fully qualified name syntax which tackles this issue in a more general setting. We defer to that proposal for the core mechanism and assume that we somehow have a way to fully qualify names using the module name. The question then becomes: what module name should be used for a definition for declarations that come from the cross-import overlay? We suggest that the name of the primary underlying module be used instead.

Going back to our running example, say all of NumericKit, FormatKit and _NumericKitFormatKitAdditions have a function named blop() (yes, I realize the example is a bit contrived).

// Say the accepted syntax is using '::' for module qualification instead of '.'

import NumericKit
import FormatKit
// import _NumericKitFormatKitAdditions // overlay imported implicitly

blop()             // error: ambiguous call to blop()
FormatKit::blop()  // works
NumericKit::blop() // works: refers to the declaration from the overlay

One might wonder: “how can you refer to the declaration from NumericKit"? It’s not possible to do that. We anticipate that it is extremely unlikely that the owners of the cross-import overlay (they would also own the primary underlying module) — would want to use the same name in both modules and have them both be usable. The simplest way to do so is giving the two declarations different names from the beginning.

Interaction with other kinds of imports

  • @_exported imports: The cross-import overlay is @_exported-imported if all the underlying modules are also @_exported-imported.

  • @_implementationOnly imports: The cross-import overlay is @_implementationOnly-imported if any of its underlying modules is @_implementationOnly-imported.

Alternatives

Maintain the status quo

As argued in the beginning, this is not a good state of affairs as it prevents framework authors from providing a natural home for cross-cutting functionality.

Have a single overlays.json instead of individual JSON files

One might ask: why have so many individual JSON files, one per overlay, instead of having one combined Overlays.json file that describes all of them? One reason to have separate files is that in a build system, the latter avoid race conditions if multiple jobs happen to modify the same Overlays.json file at once.

Use special feature flags to enable extra functionality

For example, one could have a framework level feature flag that is strictly additive. Otherwise, if the flag can remove definitions, and if one happens to transitively depend on the framework with both the flag on and off, one would need both copies of the framework.

We think this solution is not as good as our approach outlined above because:

  1. It requires more implementation work: we would need to come up with a design for strictly additive feature flag and the implementation would probably touch many different parts of the compiler.
  2. It increases the amount of work the compiler needs to do: we would need to make sure that the framework compiles successfully both with the flag on and off.
  3. It will lead to poorer build parallelism and incrementality: (assuming library evolution is enabled) changes to the flag-imported framework might end up rebuilding the whole flag-using framework (which might be large), instead of just rebuilding the overlay (which would probably be smaller).

We think the cross-import overlays approach is cleaner both in terms of implementation and likely to lead to better outcomes for build times.

Concerns and Limitations

How will ambiguities be resolved in case there are multiple possible candidates?

Say somehow we end up having two modules _NumericKitFormatKitAdditions and _FormatKitNumericKitAdditions — how will the compiler pick which one to import? Will it import both? Will it import neither?

We think that if both of these modules are present, it is probably a bug, either in some code or in the communication between framework authors. We don't think there is a good engineering reason for such ambiguous cases to arise in practice. If it does, the compiler will emit a note (not a warning) asking the developer to contact the framework authors about the situation, and not implicitly add any import statement.

The downstream developer may still write out import _NumericKitFormatKitAdditions (or import _FormatKitNumericKitAdditions , or both) if they wish to do so.

Will my framework be able to provide retroactive cross-import overlays, i.e., overlays for which all underlying modules are third party modules?

No. For example, if you work at Capitalist Enterprise Inc., and your CapitalistEnterprise.framework contains a module StockKit, then it can provide an overlay for StockKit + NumericKit. However, it cannot provide an overlay for NumericKit + FormatKit. Allowing such overlays makes it more confusing for downstream clients (adding a new, seemingly unused framework somehow changed the build, huh?), and increases the I/O the compiler needs to do because such retroactive overlays could be in any framework.

Will this adversely affect diagnostics and autocomplete in an IDE?

Since the exact naming scheme will be decided up front, the compiler's diagnostics, and the information it gives to downstream IDE clients can be adjusted to provide more information on whether a declaration came from a cross-import overlay or not.

Similarly for autocomplete -- we expect that exact name of the cross-import overlay will be hidden in most cases but one could perhaps, if one wanted, ask for it explicitly.

We expect the behavior here to be the same as that for the existing overlays.

Will I be able to turn this off if I don't want these invisible imports?

We do not expect many people to want to turn off these automatic imports. So the initial version of the feature will not have the ability to turn it off. If usage reveals that, yes, this is indeed something desired by many developers, we can add an attribute to achieve this functionality, say something like (strawman syntax):

@skipOverlay("FormatKit")
import NumericKit

While we don't want people to do this, pedantically speaking, it would be possible to manually pass framework/include paths to contrive a situation in which the cross-import overlay is not present on the search path. In such a situation, the compiler will emit a note about the missing overlay, not a hard error.

Will this create additional challenges for my company's build system?

We don't think it will. The design is very deliberate in that the paths where swiftmodules are located is kept unchanged, so that the caching logic in a build system doesn't need to be changed. The only change is to how the compiler figures out which module needs to be imported.

The compiler's emitted dependencies will certainly contain paths to the cross-import overlays that were imported, as one might expect.

Will these replace the existing overlays?

Conceptually, the existing overlays may be thought of as cross-import overlays with the primary underlying module being the standard library. Since the standard library is implicitly imported by every module, the effect is that adding a single import gets you the overlay as well.

It would take some effort to make sure that this works well in practice without any gotchas’. So for now, the existing overlays will continue to exist as they are. In the future, we might change that if we are able to do so in a source-compatible and binary-compatible way.

Will I be able to "sink" code from a module to a cross-import overlay?

No, you can't do that, as that would break source compatibility for clients who only import one of the underlying modules. It would also break binary compatibility as the linker would not know that the cross-import overlay library also needs to be linked.

More generally, it is not possible to move code from a module to one of its dependents without breaking compatibility.

What about submodules?

Submodules do not solve the same problems that cross-import overlays do, so they are not an alternative to cross-import overlays. It is hard to speculate about features for which there are no active pitches/proposals — we anticipate that if submodules are added in the future, this design will still work without many compromises.

9 Likes
2

This seems like a really nice improvement, thanks for writing it up!

My one concern right now is around what should happen if the user tries to import an overlay directly. For example, if I write import _NumericKitFormatKitAdditions, should the overlay declarations shadow those in both the primary and secondary modules, or should it be equivalent to writing import NumericKit; import FormatKit? Ideally, I think importing an overlay directly would be disallowed by the compiler instead of just being discouraged by convention by using the underscore prefix. That would require adding some kind of flag or marker to the overlay module itself though.

3

I actually think it’s good that you can import the module directly. If you’re using a Swift 5.1 compiler (which of course won’t have this feature), you can explicitly import the overlay module to get its APIs. Once you upgrade, that explicit import becomes a harmless no-op.

1 Like
4

It should behave identically in the cases where was imported implicitly vs imported explicitly. Behavior that differs from that has two downsides (1) it makes testing harder and (2) one more edge case that needs to be remembered. I don't see any benefits of making the behavior different. Do you have some in mind?

@beccadax mentioned that this is a good thing because it makes migrating easier; you can already start using the feature without having a compiler which has this built-in. There are other benefits too:

  1. Testing the feature is easier.
  2. If there is a compiler bug where the overlay is not getting imported due to some interplay of factors, you can still spell it out and get your work done instead of getting blocked on the bug fix.
  3. We will want to write these explicitly in a swiftinterface so that the compiler can bypass the overlay lookup when compiling from a swiftinterface. If we disallow writing it explicitly, that means that we either (a) redo the overlay lookup when compiling from a swiftinterface OR (b) have a divergence between what you can do in the source code vs in a swiftinterface.
2 Likes
5

I hadn't considered this, it's an excellent reason to allow manual imports along with @typesanitizer's point about avoiding divergence in swiftinterfaces.

No, and I agree the behavior should be identical. I could see a situation where it might be nice to have overlays implicitly re-export the modules they overlay, but I think that's probably a bad idea.

6

I'm not really seeing the problem. The external libraries I use "sometimes" have a second dependency that I need to also import. This is no burden. The dependency managers already have their own syntax for pulling in implicit dependencies so no new JSON syntax seems needed. Maybe you're worried about RAM being taken up by unused code? That's the responsibility of the VM system to solve and maybe the dynamic linker.

Maybe I don't understand what cross-cutting means in this context but I don't see the problem that's being solved.

3 Likes
7

I agree. I mean, I can kinda see the problem - you're not sure if you should put the extensions in module A or module B. But making a whole new module C, which is a new kind of module which is implicitly imported if both A and B are, just seems like overkill.

People who import A and B will see the extensions anyway - so, I mean... does it really matter? What we do now is very easy to reason about, and gives you access to implementation details of A or B (depending which framework you decide to put them in).

This change would be harder to reason about and end up with access to neither A nor B's implementation details. And for what?

3 Likes
8

you're not sure if you should put the extensions in module A or module B

It is not just about extensions. If a class from one module depends on a protocol from another module, and you want to provide a conformance, then:

  1. You write a retroactive conformance: People really shouldn't be writing these. Retroactive Conformances vs. Swift-in-the-OS
  2. One of the package depends on the other: This introduces build ordering issues as the compilation must proceed one after the other instead of in parallel. Moreover, in the case these are not system frameworks, people who only want one package need to compile more things which they are not using.

But making a whole new module C, which is a new kind of module which is implicitly imported if both A and B are, just seems like overkill.

These are a generalization of the overlays that we already have today. Moreover, you can use them like any other module with an explicit import _NumericKitFormatKitAdditions in the source if you wish to do so.

People who import A and B will see the extensions anyway - so, I mean... does it really matter?

I don't follow. Are you proposing that the solution one should pick in the above example is that NumericKit should depend on FormatKit (or vice-versa)?

What we do now is very easy to reason about, and gives you access to implementation details of A or B (depending which framework you decide to put them in).
This change would be harder to reason about and end up with access to neither A nor B's implementation details.

If you need access to the implementation details, you can still put that code in the corresponding module where the implementation resides. This change is additive, it does not take away that freedom.

And for what?

@phoneyDev also echoed a similar sentiment.

I don't see the problem that's being solved.

Let me reiterate the key benefits of this approach:

  1. Build ordering: Using cross-import overlays provides greater build parallelism (and potentially less compilation work) compared to having package depend on the other. This is particularly beneficial if you are building from source.
  2. No worries about retroactive conformances (and generally cross-cutting functionality): The cross-import overlay provides a natural home for this functionality.
  3. Good defaults + opt-in explicitness: You get the functionality of both frameworks if and only if you import both of them [same as today] but without additional clutter. On the other hand, if you want your code to explicitly list each and every import, you can do that too.
1 Like
9

How does canImport fit into this?

10

canImport works exactly the same as before. Because you can look up this third module by name, you can also #if canImport(_NumericKitFormatKitAdditions).

11

I guess what I mean is… shouldn’t canImport be the starting point for this? It seems like logically that’s the existing and appropriate way to conditionally expose / add functionality based on the opportunistic availability of other modules… I keep trying to figure if this proposal is really just about the mechanics of binary module distribution within that language framework… but it doesn’t seem written as if that’s the case; it seems written as if it’s a mostly parallel thing. So I’m wondering if I misunderstand what this is trying to do and/or what canImport already does…?

1 Like
12

There's a difference between whether the compiler is able to import a module and whether it does import a module. The implication here is not "if NumericKit is available, import that and add this stuff", the question is "If the client has already imported NumericKit, add this extra stuff on top to make it work better with FormatKit".

The core idea being if you haven't imported NumericKit, even if you can, you don't incur the costs of loading it and linking against it at runtime.

1 Like
13

Great write-up! I have several questions to clarify.

Shouldn't _NumericKitFormatKitAdditions.framework be a top-level framework just like NumericKit and FormatKit? The example in the pitch looks like _NumericKitFormatKitAdditions is nested in NumericKit:

# new overlay
NumericKit.framework/Modules/_NumericKitFormatKitAdditions.swiftmodule/Overlays/TestKit.json

We recommend using _ as prefix for these implicitly imported modules because we want to hide them in the code completion results, correct? Is this sufficient? cc: @akyrtzi and @rintaro

14

It could be a top-level framework (as in your example), or it could be another module provided by the same framework (as in my example).

I don't see a reason why we would want to enforce that it must be a top-level framework.

One key invariant we'd like to have is that the lookup rules for the overlay swiftmodule should not differ from the lookup rules for ordinary swiftmodules.

I'm not super familiar with how our lookup rules interact with framework paths passed using -F -- I guessed they work similar to paths passed using -I but we look at the Modules subdirectory instead. If my guess is correct, and if we say "it must be a top-level framework" then I don't think we can maintain the invariant mentioned earlier.

The _ specifically was intended as a hint to both a radar and for code completion. Code completion can potentially do one of several things here:

  • check for just _ (potentially bad if people are using it for their own internal things)
  • check for the _ and Additions at the end (more robust)
  • I'm guessing SourceKit gets the available modules from the compiler... if so, the compiler API can be tweaked to also provide more information about whether the module was a cross-import overlay and then SourceKit/IDE can decide what they'd like to do with it.
15

There's a "says who?" problem here. Dissatisfying to whom? Displeasing to whom? There's insufficient evidence of need for a change or addition to the language.

The warning sign is that the whole pitch is based on a single fictional example. Could the problem be solved by choosing more elegant names than NumericKitFormatKitAdditions etc? Or, are there any real-world examples that demonstrate a problem that many people actually run into, that has no reasonable solution? Enough people to justify adding a new opaque feature into the language?

3 Likes
16

AFAIK (someone correct me if I'm wrong), SerializedModuleLoaderBase currently requires that the framework and module basenames match, so you could only load _NumericKitFormatKitAdditions from _NumericKitFormatKitAdditions.framework/Modules/_NumericKitFormatKitAdditions.swiftmodule.

One could propose expanding that logic to allow other kinds of searches, but I think a big problem would be efficiency; if you allow a framework to contain modules with names other than the framework name, you can't just check each framework search path for the existence of $NAME.framework; you have to walk all the *.frameworks on all search paths until you find one containing Modules/$NAME.swiftmodule (or your proposed overlay).

17

It's about avoiding pulling in dependencies if there isn't a need for them. Imagine there's a really great new networking framework called NetworkKit, and ReactiveSwift wants to add special bindings for NetworkKit. If they want to do that, they have two options:

  • Create a new module, ReactiveNetworkKit, and require people to import it separately
  • Import NetworkKit from ReactiveSwift and make it a dependency

Creating a new module is probably the most practical approach, which is why ReactiveSwift chose ReactiveCocoa and made it depend on ReactiveSwift. But if I have ReactiveCocoa and ReactiveNetworkKit, I still have to duplicate the imports everywhere. And every new "overlay" they add, I have to import myself.

Making ReactiveSwift depend on NetworkKit is also a bad option, because if I'm writing a program that has nothing to do with networking, I shouldn't have to bring in, distribute, and link an entire new dependency in.

This feature would allow frameworks to augment other frameworks without introducing new dependencies on all of their clients, and that's the value it brings.

5 Likes
18

Thanks for pointing this out. I double-checked this with others and looks like I misunderstood the usage. @Xi_Ge in that case, you're right, the framework directory needs to match the swiftmodule name. [I'll fix the pitch.]

19

Makes sense.

But then why not have a natural variation of canImport, e.g. willImport? Even if it can only be applied at certain points (if / by necessity), e.g. in the ‘header’ of an extension block.

This pitch seems to gloss over the details of how the actual code is structured / affected by this proposal, which I think it should elaborate on. It seems to be enforcing essentially the existing approach of creating a [potentially large] number of independent frameworks, combinatorially - it’s merely adding a fairly thin sprinkling of sugar to what is fundamentally a scale-challenged and awkward approach. IMO it’s much more natural to treat these conditional extensions just like conditional conformance in protocols and the like. And if you don’t like that ‘intertwined’ style, you can choose to partition things explicitly into separate modules - but you’re not forced to.

2 Likes
20

I think the pitch did a good enough job explaining the reasoning for not picking this:

Another thing to consider is that #if blocks are only meant to affect the module at the time it is compiled, and intentionally not downstream clients of a module. We explicitly strip out #if blocks from .swiftinterface files for this reason. Granted, this doesn't necessarily need to be a #if condition, but it's not just a purely natural extension of canImport.

This proposal was written such that it doesn't affect how the code is structured, beyond adding an additional implicit import when two other modules are imported.

I really don't think this is as scale-challenged as you're portraying it to be. These overlay modules will rarely, if ever, be more than one level deep.

2 Likes