Cross-Import Overlays

#8

you're not sure if you should put the extensions in module A or module B

It is not just about extensions. If a class from one module depends on a protocol from another module, and you want to provide a conformance, then:

  1. You write a retroactive conformance: People really shouldn't be writing these. Retroactive Conformances vs. Swift-in-the-OS
  2. One of the package depends on the other: This introduces build ordering issues as the compilation must proceed one after the other instead of in parallel. Moreover, in the case these are not system frameworks, people who only want one package need to compile more things which they are not using.

But making a whole new module C, which is a new kind of module which is implicitly imported if both A and B are, just seems like overkill.

These are a generalization of the overlays that we already have today. Moreover, you can use them like any other module with an explicit import _NumericKitFormatKitAdditions in the source if you wish to do so.

People who import A and B will see the extensions anyway - so, I mean... does it really matter?

I don't follow. Are you proposing that the solution one should pick in the above example is that NumericKit should depend on FormatKit (or vice-versa)?

What we do now is very easy to reason about, and gives you access to implementation details of A or B (depending which framework you decide to put them in).
This change would be harder to reason about and end up with access to neither A nor B's implementation details.

If you need access to the implementation details, you can still put that code in the corresponding module where the implementation resides. This change is additive, it does not take away that freedom.

And for what?

@phoneyDev also echoed a similar sentiment.

I don't see the problem that's being solved.

Let me reiterate the key benefits of this approach:

  1. Build ordering: Using cross-import overlays provides greater build parallelism (and potentially less compilation work) compared to having package depend on the other. This is particularly beneficial if you are building from source.
  2. No worries about retroactive conformances (and generally cross-cutting functionality): The cross-import overlay provides a natural home for this functionality.
  3. Good defaults + opt-in explicitness: You get the functionality of both frameworks if and only if you import both of them [same as today] but without additional clutter. On the other hand, if you want your code to explicitly list each and every import, you can do that too.
1 Like
#9

How does canImport fit into this?

#10

canImport works exactly the same as before. Because you can look up this third module by name, you can also #if canImport(_NumericKitFormatKitAdditions).

#11

I guess what I mean is… shouldn’t canImport be the starting point for this? It seems like logically that’s the existing and appropriate way to conditionally expose / add functionality based on the opportunistic availability of other modules… I keep trying to figure if this proposal is really just about the mechanics of binary module distribution within that language framework… but it doesn’t seem written as if that’s the case; it seems written as if it’s a mostly parallel thing. So I’m wondering if I misunderstand what this is trying to do and/or what canImport already does…?

1 Like
#12

There's a difference between whether the compiler is able to import a module and whether it does import a module. The implication here is not "if NumericKit is available, import that and add this stuff", the question is "If the client has already imported NumericKit, add this extra stuff on top to make it work better with FormatKit".

The core idea being if you haven't imported NumericKit, even if you can, you don't incur the costs of loading it and linking against it at runtime.

#13

Great write-up! I have several questions to clarify.

Shouldn't _NumericKitFormatKitAdditions.framework be a top-level framework just like NumericKit and FormatKit? The example in the pitch looks like _NumericKitFormatKitAdditions is nested in NumericKit:

# new overlay
NumericKit.framework/Modules/_NumericKitFormatKitAdditions.swiftmodule/Overlays/TestKit.json

We recommend using _ as prefix for these implicitly imported modules because we want to hide them in the code completion results, correct? Is this sufficient? cc: @akyrtzi and @rintaro

#14

It could be a top-level framework (as in your example), or it could be another module provided by the same framework (as in my example).

I don't see a reason why we would want to enforce that it must be a top-level framework.

One key invariant we'd like to have is that the lookup rules for the overlay swiftmodule should not differ from the lookup rules for ordinary swiftmodules.

I'm not super familiar with how our lookup rules interact with framework paths passed using -F -- I guessed they work similar to paths passed using -I but we look at the Modules subdirectory instead. If my guess is correct, and if we say "it must be a top-level framework" then I don't think we can maintain the invariant mentioned earlier.

The _ specifically was intended as a hint to both a radar and for code completion. Code completion can potentially do one of several things here:

  • check for just _ (potentially bad if people are using it for their own internal things)
  • check for the _ and Additions at the end (more robust)
  • I'm guessing SourceKit gets the available modules from the compiler... if so, the compiler API can be tweaked to also provide more information about whether the module was a cross-import overlay and then SourceKit/IDE can decide what they'd like to do with it.
#15

There's a "says who?" problem here. Dissatisfying to whom? Displeasing to whom? There's insufficient evidence of need for a change or addition to the language.

The warning sign is that the whole pitch is based on a single fictional example. Could the problem be solved by choosing more elegant names than NumericKitFormatKitAdditions etc? Or, are there any real-world examples that demonstrate a problem that many people actually run into, that has no reasonable solution? Enough people to justify adding a new opaque feature into the language?

3 Likes
#16

AFAIK (someone correct me if I'm wrong), SerializedModuleLoaderBase currently requires that the framework and module basenames match, so you could only load _NumericKitFormatKitAdditions from _NumericKitFormatKitAdditions.framework/Modules/_NumericKitFormatKitAdditions.swiftmodule.

One could propose expanding that logic to allow other kinds of searches, but I think a big problem would be efficiency; if you allow a framework to contain modules with names other than the framework name, you can't just check each framework search path for the existence of $NAME.framework; you have to walk all the *.frameworks on all search paths until you find one containing Modules/$NAME.swiftmodule (or your proposed overlay).

#17

It's about avoiding pulling in dependencies if there isn't a need for them. Imagine there's a really great new networking framework called NetworkKit, and ReactiveSwift wants to add special bindings for NetworkKit. If they want to do that, they have two options:

  • Create a new module, ReactiveNetworkKit, and require people to import it separately
  • Import NetworkKit from ReactiveSwift and make it a dependency

Creating a new module is probably the most practical approach, which is why ReactiveSwift chose ReactiveCocoa and made it depend on ReactiveSwift. But if I have ReactiveCocoa and ReactiveNetworkKit, I still have to duplicate the imports everywhere. And every new "overlay" they add, I have to import myself.

Making ReactiveSwift depend on NetworkKit is also a bad option, because if I'm writing a program that has nothing to do with networking, I shouldn't have to bring in, distribute, and link an entire new dependency in.

This feature would allow frameworks to augment other frameworks without introducing new dependencies on all of their clients, and that's the value it brings.

5 Likes
#18

Thanks for pointing this out. I double-checked this with others and looks like I misunderstood the usage. @Xi_Ge in that case, you're right, the framework directory needs to match the swiftmodule name. [I'll fix the pitch.]

#19

Makes sense.

But then why not have a natural variation of canImport, e.g. willImport? Even if it can only be applied at certain points (if / by necessity), e.g. in the ‘header’ of an extension block.

This pitch seems to gloss over the details of how the actual code is structured / affected by this proposal, which I think it should elaborate on. It seems to be enforcing essentially the existing approach of creating a [potentially large] number of independent frameworks, combinatorially - it’s merely adding a fairly thin sprinkling of sugar to what is fundamentally a scale-challenged and awkward approach. IMO it’s much more natural to treat these conditional extensions just like conditional conformance in protocols and the like. And if you don’t like that ‘intertwined’ style, you can choose to partition things explicitly into separate modules - but you’re not forced to.

1 Like
#20

I think the pitch did a good enough job explaining the reasoning for not picking this:

Another thing to consider is that #if blocks are only meant to affect the module at the time it is compiled, and intentionally not downstream clients of a module. We explicitly strip out #if blocks from .swiftinterface files for this reason. Granted, this doesn't necessarily need to be a #if condition, but it's not just a purely natural extension of canImport.

This proposal was written such that it doesn't affect how the code is structured, beyond adding an additional implicit import when two other modules are imported.

I really don't think this is as scale-challenged as you're portraying it to be. These overlay modules will rarely, if ever, be more than one level deep.

2 Likes
#21

It's sufficient in the current implementation. For module name completion (after import), we hide all modules starting with _.

1 Like
#22

My apologies if I missed it in my reading but is there a rationale for not 'just' explicitly stating the dependencies in the json file and then generate an underscored name? This might be a complete non-issue but it makes more sense to me somehow.

{
  "version": 1,
  "parents": [ // absolutely no strong feelings about the name of this key
    {"name": "NumericKit"},
    {"name", "FormatKit"}
  ]
}
#23

This functionality would be very, very welcome.

Our existing (single-module) overlays are shipping as regular standalone dylib targets (not frameworks). Discovery is done through a naming scheme, with no requirement for explicit registration. Is there a specific reason to define these cross-cutting overlays as full frameworks, or was that just a non-normative example? (Our existing single-module overlays are standalone dylibs, not frameworks.)

+1. I think we should have a strictly enforced naming scheme for such overlay modules.

Followup: why not just do away with JSON descriptors entirely, and figure out what cross-cutting overlays need to be loaded based on their name/location, like we do for regular overlays? (There is a combinatorial explosion, but only if we don't have a complete list of all such overlays. Admittedly I have no idea if building such a list is feasible.)

#24

That does seem nicer but one problem with that is now the compiler (and any other tool that looks at the JSON file) now needs to know how to compute the name of the cross-import overlays from the names of the parents. In terms of amount of implementation work, it is not that much, but it seems an unnecessary addition; now the naming scheme is part of the Swift language instead of being a convention.

I suppose it is a matter of opinion whether that is a big deal or not. Other things being equal, I would prefer that we not wire more of these little details into the compiler. If that helped out users in a major way, that would change the equation but I don't think this qualifies.

But yeah, since this is a small detail, I can change it if other people feel the same way. :slight_smile:

#25

I actually think that the language holding it is an advantage but I admit that it is mostly my feeling. A stable order/naming that Swift managed seems like the best option to me.

#26

I think it would be important to be able to look at a .swiftmodule and understand from a glance that it is supposed to be a combo overlay, not a standalone module. Enforcing consistent naming is a good way to ensure that.

The problem with allowing people to customize the cross-import module name is, of course, that people will then go ahead and customize the module name.

#27

Is there a specific reason to define these cross-cutting overlays as full frameworks, or was that just a non-normative example?

The idea behind making them a full framework is that the author can avail of the flexibility of frameworks. For example, if you want to have some data files/assets associated with the overlay, it might make more sense to use a framework compared to embedding those assets in the dylib itself.

Followup: why not just do away with JSON descriptors entirely, and figure out what cross-cutting overlays need to be loaded based on their name/location, like we do for regular overlays? (There is a combinatorial explosion, but only if we don't have a complete list of all such overlays. Admittedly I have no idea if building such a list is feasible.)

The problem with building such a list is that it is bad from a caching perspective. Adding a single module anywhere along any search path invalidates the cache, which is quite pessimistic. We already have this problem today in the form of -I today... having this kind of lookup would make the problem worse.

2 Likes
Terms of Service

Privacy Policy

Cookie Policy