Update on implementation-only imports

#1

After the long discussion on the previous thread, @John_McCall and I went away to go prototype the main new feature: implementation-only imports. (I'm staying away from "private imports" to avoid the implications of access control for now, as well as the other experimental "private imports" feature that's in support of dynamic method replacement.) We've been doing this work on the master and 5.1 branches of the compiler so that some Apple projects can live on it for a while to make sure (1) it works, and (2) the design is actually what we want. Note that it's not going to be an official feature of Swift 5.1.

Meanwhile, if you want to play around with it, a reminder that this is unsupported, possibly buggy, and definitely not the final syntax. But other than that, you're welcome to!

Right now the new feature looks like this:

// MyKit.swift
@_implementationOnly import SecretKit

The name is pretty unwieldy right now since we expect that'll be a hot topic of discussion. (We've gotten some feedback that it'd be worth trying to change the default semantics of import; I'm personally skeptical whether we can do this in a non-breaking way.)

What does this do today?

  • Clients of MyKit will not need SecretKit around when using MyKit. SecretKit won't even be loaded, avoiding the problems listed in "@_exported and fixing import visibility".

  • The public API and ABI of MyKit is not allowed to use any declarations from SecretKit. This is enforced by the compiler.

  • The public API and ABI of MyKit is not allowed to rely on conformances declared in SecretKit, even if the types involved are from other modules. This is subtle but probably won't come up in practice. (Also enforced by the compiler.)

  • The public types of MyKit are not allowed to conform to protocols in SecretKit. We consider this a limitation, but it's a tricky restriction to lift given the way the several parts of the compiler work right now.

  • If a public type in MyKit overrides a method defined in SecretKit, the override has to be marked @_implementationOnly as well, so that it can be omitted from the module interface. This feels quite hacky to me; everywhere else things are enforced by the compiler, but here the compiler mostly just trusts the developer to get it right once they've put the attribute on.

  • A generated module interface (.swiftinterface file) will not make any mention of SecretKit, and should not reference anything from SecretKit either.

  • The compiled module (.swiftmodule file) may still have references to SecretKit for debugging and testing purposes. This means that the compiler must know not to rely on the parts that depend on SecretKit when the compiled module is used for building clients. (This is where 90% of the known issues are right now.)

  • @testable import MyKit is probably completely broken. This should probably bring in the implementation-only dependencies of MyKit even though they normally wouldn't be brought in, since the alternative is to figure out what's being used in MyKit's internal types as well.

  • LLDB support is only verrrry lightly tested. (I haven't even managed to land the PR with the tests in it, mostly because I've been busy with other things.)

  • I've tried not to make anything dependent on -enable-library-evolution, because I know people want this for packages, but the adopters within Apple are all using library evolution support, so it wouldn't surprise me if things are worse in "normal" / "fragile" mode.

Again, this is what's currently implemented, not necessarily what we want to propose in the long run. And it doesn't cover a number of other use cases, particularly vending content that's only present if the client asks for it.

24 Likes
@_exported and fixing import visibility
Building .swiftmodules for SourceKit
Swift packages and module dependencies
Two Libraries with Shared Source...Overlapping Error
Backwards-deployable Conformances
Unknown attributes should be a warning (not an error)
#2

Probably also worth mentioning that implementation-only import won't affect symbol mangling and you still can't have two modules named SecretKit that end up in the same "package graph".

#3

Sure, right. It's a modifier on the import only; it doesn't affect the existence or contents of the module being imported.

#4

How a public type can override a method without confirming to a SecretKit protocol or being able to inherit a SecretKit type ?

[edit] I just read the PR link description. This is about overriding methods declared in extension of existing type inside SecretKit [/edit]

1 Like
#5

I don’t understand the why. What is the purpose of this feature?

#6

See the thread @_exported and fixing import visibility for more information.

If you have a dependency that is entirely an implementation detail, such as how you do JSON parsing - I could see this being useful.

It also gives compiler support for enforcing the intended use of the dependency.

1 Like
#7

Yep, @Mordil's got it. There are times where a dependency should not be exposed to clients, whether that's because of Swift's current leaky imports that we hope to fix at some point (@_exported and fixing import visibility), or because you're making something for binary distribution and won't be distributing your dependencies. That latter is Apple's main use case (public frameworks being implemented in terms of private frameworks), but it's also a step in addressing one of the biggest limitations in the Xcode 11 binary framework support: you can't use packages with it, because those become dependencies for your clients as well.

(I compressed a lot into that one sentence; check out the WWDC session I co-presented. The problem here is talked about at 37:16, albeit briefly.)

Now, this alone doesn't solve all the problems there, because you still can't have two copies of the same package in the same address space. But it's a step, and it would at least allow for local packages, prefixed with your organization's name in some way, to be used in XCFrameworks. As long as you're careful about not packaging them into two separate binary frameworks, anyway.

In general we really do want to improve the cross-module dependency story in Swift, both for source packages and for binary packages. We as library authors should be able to separate compilation dependencies from link dependencies and control how both are passed on to a library's clients.

(sorry for not responding sooner)

7 Likes
#8

Here is something that has a difference between -enable-library-evolution and not.

If you have:
something.h:

struct A { ... };

something.swift:

struct B {
  var v: UnsafeMutablePointer<A>;
};

With -enable-library-evolution it works, but without it complains with: "error: cannot use struct 'A' ... has been imported as implementation-only"

1 Like
#9

I just wrote a post on my experiences splitting a package into modules (Swift packages and module dependencies ) where one of the issues I raised was automatic "re-export" of conformances and extensions added to imported public types. My suggestion was

but hopefully this feature will deal with this issue!

1 Like
#10

Any news on this? We really need this feature :(

1 Like
#11

@zienag Jordan Rose is not at Apple anymore maybe someone else at Apple will continue his work maybe not, we shall see.

1 Like
#12

The core work is already present in the compiler; it's just (unfortunately) all using internal, non-stable language features.

3 Likes
#13

One thing we noticed when using this feature is that you're required to match imports across the entire module with or without @_implementationOnly. We are thinking that this could lead to a cumbersome case where a module A with 50 files (and therefore up to 50 imports) depends on B as implementation only, but then A wants to expose B in its public interface, and the developer has to remove the annotation from all @_implementationOnly import Bs. Is there any interest in adding a compiler level flag that treats all "normal" import B declarations as @_implementationOnly import B instead, so therefore doing this transition would be a matter of removing that flag? Ignore the spelling but something like -implementation-only-import B

2 Likes
#14

To add to Keith's comment, I was also wondering about the possibility of being able to use per-file @_implementationOnly imports. In other words mixed use of regular imports and @_implementationOnly imports within a module. If this was allowed, people could see exactly which files were exposing (or leaking) dependencies. Seeing this info could help for maintenance and module structuring.

EDIT: It's a warning right now to used mixed imports, which means it's allowed if you don't use -warnings-as-errors, but we do use that. A case where control over warnings could be nice.

#15

I've submitted a PR for discussion that allows the example that @Dave_Lee is suggesting https://github.com/apple/swift/pull/29230

#16

This was a deliberate decision, actually. I had originally implemented it the way you described, but we immediately found that it made it too easy to accidentally publish the dependency on B without the compiler telling you anything. In the long term, perhaps "new dependencies" should be part of the purview of the API checker (@Xi_Ge, who should that idea go to?) and then the compiler can switch / go back to allowing mixed use of implementation-only.

2 Likes
#17

Thanks for the context! Do you have any thoughts on something like this:

#18

As a general principle, I don't like module-wide command-line flags that affect the interpretation of source code, because it means you can't look at the source code in isolation and know what it's doing. But I haven't really taken stock of what the current set of flags are these days (-enable-library-evolution is doing exactly that, for example), or thought about whether this is a case where it makes sense.

I do know that the core team was interested in changing the default. I personally think that's going to be intractable—everyone using types from Foundation in their public APIs is going to have to learn about this new kind of import. So maybe an "increased paranoia level" flag makes sense here.

(I also know package manager folks have long been interested in making dependencies more explicit, so it might be good to get them into the discussion.)

1 Like
Terms of Service

Privacy Policy

Cookie Policy