After the long discussion on the previous thread, @John_McCall and I went away to go prototype the main new feature: implementation-only imports. (I'm staying away from "private imports" to avoid the implications of access control for now, as well as the other experimental "private imports" feature that's in support of dynamic method replacement.) We've been doing this work on the master and 5.1 branches of the compiler so that some Apple projects can live on it for a while to make sure (1) it works, and (2) the design is actually what we want. Note that it's not going to be an official feature of Swift 5.1.
Meanwhile, if you want to play around with it, a reminder that this is unsupported, possibly buggy, and definitely not the final syntax. But other than that, you're welcome to!
The name is pretty unwieldy right now since we expect that'll be a hot topic of discussion. (We've gotten some feedback that it'd be worth trying to change the default semantics of import; I'm personally skeptical whether we can do this in a non-breaking way.)
What does this do today?
Clients of MyKit will not need SecretKit around when using MyKit. SecretKit won't even be loaded, avoiding the problems listed in "@_exported and fixing import visibility".
The public API and ABI of MyKit is not allowed to use any declarations from SecretKit. This is enforced by the compiler.
The public API and ABI of MyKit is not allowed to rely on conformances declared in SecretKit, even if the types involved are from other modules. This is subtle but probably won't come up in practice. (Also enforced by the compiler.)
The public types of MyKit are not allowed to conform to protocols in SecretKit. We consider this a limitation, but it's a tricky restriction to lift given the way the several parts of the compiler work right now.
If a public type in MyKit overrides a method defined in SecretKit, the override has to be marked @_implementationOnly as well, so that it can be omitted from the module interface. This feels quite hacky to me; everywhere else things are enforced by the compiler, but here the compiler mostly just trusts the developer to get it right once they've put the attribute on.
A generated module interface (.swiftinterface file) will not make any mention of SecretKit, and should not reference anything from SecretKit either.
The compiled module (.swiftmodule file) may still have references to SecretKit for debugging and testing purposes. This means that the compiler must know not to rely on the parts that depend on SecretKit when the compiled module is used for building clients. (This is where 90% of the known issues are right now.)
@testable import MyKit is probably completely broken. This should probably bring in the implementation-only dependencies of MyKit even though they normally wouldn't be brought in, since the alternative is to figure out what's being used in MyKit's internal types as well.
LLDB support is only verrrry lightly tested. (I haven't even managed to land the PR with the tests in it, mostly because I've been busy with other things.)
I've tried not to make anything dependent on -enable-library-evolution, because I know people want this for packages, but the adopters within Apple are all using library evolution support, so it wouldn't surprise me if things are worse in "normal" / "fragile" mode.
Probably also worth mentioning that implementation-only import won't affect symbol mangling and you still can't have two modules named SecretKit that end up in the same "package graph".
Yep, @Mordil's got it. There are times where a dependency should not be exposed to clients, whether that's because of Swift's current leaky imports that we hope to fix at some point (@_exported and fixing import visibility), or because you're making something for binary distribution and won't be distributing your dependencies. That latter is Apple's main use case (public frameworks being implemented in terms of private frameworks), but it's also a step in addressing one of the biggest limitations in the Xcode 11 binary framework support: you can't use packages with it, because those become dependencies for your clients as well.
(I compressed a lot into that one sentence; check out the WWDC session I co-presented. The problem here is talked about at 37:16, albeit briefly.)
Now, this alone doesn't solve all the problems there, because you still can't have two copies of the same package in the same address space. But it's a step, and it would at least allow for local packages, prefixed with your organization's name in some way, to be used in XCFrameworks. As long as you're careful about not packaging them into two separate binary frameworks, anyway.
In general we really do want to improve the cross-module dependency story in Swift, both for source packages and for binary packages. We as library authors should be able to separate compilation dependencies from link dependencies and control how both are passed on to a library's clients.
I just wrote a post on my experiences splitting a package into modules (Swift packages and module dependencies ) where one of the issues I raised was automatic "re-export" of conformances and extensions added to imported public types. My suggestion was
but hopefully this feature will deal with this issue!
One thing we noticed when using this feature is that you're required to match imports across the entire module with or without @_implementationOnly. We are thinking that this could lead to a cumbersome case where a module A with 50 files (and therefore up to 50 imports) depends on B as implementation only, but then A wants to expose B in its public interface, and the developer has to remove the annotation from all @_implementationOnly import Bs. Is there any interest in adding a compiler level flag that treats all "normal" import B declarations as @_implementationOnly import B instead, so therefore doing this transition would be a matter of removing that flag? Ignore the spelling but something like -implementation-only-import B
To add to Keith's comment, I was also wondering about the possibility of being able to use per-file @_implementationOnly imports. In other words mixed use of regular imports and @_implementationOnly imports within a module. If this was allowed, people could see exactly which files were exposing (or leaking) dependencies. Seeing this info could help for maintenance and module structuring.
EDIT: It's a warning right now to used mixed imports, which means it's allowed if you don't use -warnings-as-errors, but we do use that. A case where control over warnings could be nice.
This was a deliberate decision, actually. I had originally implemented it the way you described, but we immediately found that it made it too easy to accidentally publish the dependency on B without the compiler telling you anything. In the long term, perhaps "new dependencies" should be part of the purview of the API checker (@Xi_Ge, who should that idea go to?) and then the compiler can switch / go back to allowing mixed use of implementation-only.
As a general principle, I don't like module-wide command-line flags that affect the interpretation of source code, because it means you can't look at the source code in isolation and know what it's doing. But I haven't really taken stock of what the current set of flags are these days (-enable-library-evolution is doing exactly that, for example), or thought about whether this is a case where it makes sense.
I do know that the core team was interested in changing the default. I personally think that's going to be intractable—everyone using types from Foundation in their public APIs is going to have to learn about this new kind of import. So maybe an "increased paranoia level" flag makes sense here.
(I also know package manager folks have long been interested in making dependencies more explicit, so it might be good to get them into the discussion.)
Hey @jrose thanks for pushing this feature forward!
In my company we're jumping all in on Packages and this is a key feature for us, and we are a bit anxious atm since this is still not officially supported.
Since up in the thread someone said he is not at apple anymore, would anyone else have any news on this front?
We have an (XC)framework now, lets call it A.xcframework, that uses some other packages built as (XC)frameworks, eg B.xcframework and C.xcframework, that we would like to make private dependencies of A.xcframework. The reason is, packages B and C are widely used dependencies (for example packages like Moya, Realm etc), which the final app might want to use as well, and it might want to use versions of B and C that are different from those that A is built with. Is there a way of achieving what we need, and does @_implementationOnly import solves our problem (I assume it doesn't, reading the comments above)?