[Pre-Pitch] Import access control: a modest proposal

Evolution Pitches
1

Over the last few years, the import statement has been collecting unofficial, unsupported features to help manage the dependencies between libraries. We (@xymus and @beccadax) are thinking about how to stabilize some of these into officially-supported language features.

Chief among them is the @_implementationOnly attribute. An @_implementationOnly import is completely hidden from clients who import your module. This allows clients to import your module even if they do not have access to that module, so it's great for hiding libraries that you use only as an implementation detail. To make this work, though, the compiler stops you from using a declaration imported via an @_implementationOnly import in a public, open, or @usableFromInline declaration (including the function body if it's @inlinable) if that use would be visible to your clients:

@_implementationOnly import SwiftUI

public struct MyView: View {    // error: cannot use protocol 'View' here; 'SwiftUI' has been imported as implementation-only
  public var view: EmptyView {  // error: cannot use struct 'EmptyView' here; 'SwiftUI' has been imported as implementation-only
    EmptyView()                 // OK to use inside an implementation, just not in its interface
  }
}

(Note that this restriction is only really important for libraries; app targets usually don't need to make anything public.)

The most straightforward way to stabilize this feature would be to simply remove the underscore so that it would now be spelled @implementationOnly import. But we're considering a different alternative: redesigning this feature in terms of access control and switching the default behavior of the import statement. We believe this would be a better design if we were designing the language from scratch, but it has significant source compatibility impacts, so it would need to be added in a new language version mode and requires careful consideration even in that case.

We are looking for feedback from Evolution as a whole on whether this is an idea worth pursuing and whether there are additional concerns we have not yet identified, and from the Core Team specifically about whether the source compatibility impacts are likely to be too serious to consider this idea even in a new language version.

Proposed solution

We should permit the public and internal modifiers to be applied to import statements.

public import M should behave like an import M declaration in Swift 5.5. That is:

  • Your public APIs can visibly use* the public APIs of M

  • Clients of your module will also load** M

  • However, name lookups in your clients will not look in M unless your client separately imports M, with two exceptions:

    1. Members of extensions in M on types from other modules (e.g. if M has an extension String, public members of that extension will be visible)
    2. Certain unusual declarations, like operator and precedencegroup

(Note that these exceptions were never really intended behaviors; the extension one is even considered a bug. We may want to deprecate them anyway.)

* "Visibly use" means using a declaration in a place where your module's clients will need to be able to access the declaration. For instance, using a type as a parameter or result type, a generic constraint, an extension, a superclass, an associated value, etc. of a public declaration would be visible. The bodies of @inlinable declarations and the stored properties of @frozen structs (regardless of the property's access level) are also visible. Basically, if you could not use an internal declaration in a particular place, it is a "visible use".

** "Loading" means that the compiler will read a .swiftmodule or similar file from disk and use the information in it. "Importing" means that the compiler will not only load the module, but will also make some of the declarations it describes available for your code to call.

internal import M should behave like an @_implementationOnly import M declaration in Swift 5.5. That is:

  • Your public APIs cannot visibly use the public APIs of M; attempting to do so will be diagnosed as an error, with a fix-it offering to annotate the import in that file with public

  • Clients of your module do not need to be able to load M

  • Even extension members and operator-related declarations that would be visible with a public import will not be visible in your clients

(The compiler should reject open import, fileprivate import, and private import; we don't think these modifiers make much sense for import statements.)

In Swift 5 (and earlier) mode, Swift should interpret import M as public import M, preserving current behavior. In Swift 6 mode (or whatever the next version mode is), Swift should interpret import M as internal import M, making implementation-only imports the default.

Minor behavior changes vs. existing @_implementationOnly import

@testable import will treat all internal imports in the module being imported as though they are public imports. (Currently, the compiler basically ignores @_implementationOnly imports and hopes that this doesn't break anything. It often does.)

In Swift 6 mode, there will not be a warning if some files write import M and others write public import M. (Currently, there is a warning if some files write import M and others write @_implementationOnly import M.) We think this warning was necessary with @_implementationOnly because if you weren't thinking about which behavior you wanted, you would write a plain import and visibly import the module. But with this change you would now get a non-visible import when you did that, so you'd be less likely to make a mistake.

Swift 5 mode will allow you to write public import as a synonym for import and internal import as a synonym for @_implementationOnly import without warnings on any of those forms. Swift 6 mode will deprecate @_implementationOnly import and recommend plain import instead.

Why make this change?

  1. It aligns import with other declarations, which generally take access control keywords and are not publicly visible unless public (or open) is used.

  2. It reuses the existing concept of the internal keyword instead of introducing a separate concept for this feature, improving teachability. Reusing internal makes sense for this because public declarations forbid using internal-imported declarations in the same places they forbid using internal declarations. In other words, an internal import is very much like you imported the module's public declarations but marked them internal.***

  3. Swift 5's public-by-default behavior has caused difficulties for organizations with multi-project builds which have adopted @_implementationOnly to hide project-private modules. These organizations have reported that they frequently experience broken builds because someone accidentally imported a module that isn't available in other projects without using @_implementationOnly. If internal imports were the default, these mistakes would be much rarer.

  4. When compiling a client of your module, using a public import requires the compiler to perform extra work and I/O to load and process not only the dependency you're importing, but everything it publicly imports, transitively. A single unnecessary public import can thus cause the compiler to load many extra modules that it doesn't really need. If changing the default causes many imports to become internal instead of public, we should expect some build time improvements as a result.

*** One important difference is that you can conform a public type to an internal protocol (the conformance is just hidden from clients), but conforming a public type to a public protocol imported by internal import is an error. This reduces the chance that a client will declare a retroactive conformance that clashes with yours, which would not be detectable at compile time.

What source breaks will we see?

This change will cause two kinds of source breaks:

  1. Any public, open, or @usableFromInline declaration in a module that has a visible use of an imported declaration will break in Swift 6 mode. However, these breaks will be easy to detect and offer fix-its for, and thus they'll be easy to accurately mass-migrate. They also should only affect library modules, not executable modules, since executable modules rarely need to use access control levels above internal.

  2. If a library makes a once-public import internal, its clients may break if they were depending on extensions or operator-related declarations that were transitively loaded through that library. It will not be possible to detect these breaks from within the library or automatically migrate them. However, clients can solve this problem by explicitly importing the module containing the extensions or operator declarations, and once they become aware of broken clients, libraries can fix this problem by manually marking imports as public. If we deprecate these uses—especially ahead of the release of the next major language version—people can fix these issues before the behavior of imports starts to change en masse.

What other issues will we see?

Developers have been irked in the past by similar language changes (the open keyword, the @frozen attribute) where they had to make source code changes when the compiler got stricter about imports. However, those changes did not offer an automatic migration path; here, the migrator will be able to detect when an import statement's declarations are visibly used and upgrade it to a public import.

Problems like this where a binary module and its client both link different versions of the same library, but the compiler doesn't detect this because the binary module's import is not visible to its client, will be more likely to happen because more imports will be internal.

Currently, using @_implementationOnly import in a non-library-evolution module does not allow you to use imported types as nonpublic stored properties of a public struct. This is because your clients need access to the stored property's type to compute some details of its layout. This limitation is probably not acceptable for a supported language feature, so we'll need to fix it somehow. (Perhaps we can serialize a minimal description of the type into the swiftmodule file.)

What about @_exported?

We're interested in stabilizing @_exported too, but probably as an @exported attribute on public import rather than as an access control modifier in its own right. Re-exporting is too rarely needed to be a good use of public, and the analogy to the meaning of open on classes and members is extremely weak.

(But see @jrose's counter-argument that @_exported should be public and the thing this pitch calls public should be removed or sidelined.)

69 Likes
2

Yes.

That’s it. That’s the post.

8 Likes
3

+1

How do you see this intersecting with @_spi in the future?

7 Likes
4

I approve of any removal of underscore special cases.

But I feel like there are use cases for fileprivate, and open. To me @_exported import feels like an adjustment of visibility of inheritance so it lines up w/ open. public seems quite on par w/ what we have today of normal imports, and internal would be VERY nice to have as @_implementationOnly being that the scope of the internal declaration there is to the module itself. But there are times at which I want to control the scope of types such that they do not leak through implementations and a file private restriction would be welcomed in my book either spelled as fileprivate import or private import ether way feels like something that would be helpful for framework development (especially for those of us who need to work on things that do not expose internal bits to the outside world from our frameworks when it comes to layering or types).

10 Likes
5

This seems great! I'm super excited to see imports getting some love! I like the reuse of public and internal as a way to make the behavior seem a bit more familiar for folks since they're already used to those keywords.

I think it would be a great time to fix this if it wouldn't be asking too much, assuming that folks could pretty easily recover from that, and they would have to have opted in to Swift 6 mode anyways. There's a big thread on potential less major solutions to this but I haven't gotten a chance to actually work on any of them Supporting strict imports

I think so too, from our attempts to use @ _implementationOnly in a modularized project for the same reasons you described this specific issue meant we can't use them nearly anywhere in our project because almost every module is included for some type you want to store at some point.

I think it's still worth considering a warning for this case because one thing I could see happening is a developer not realizing that another file in the library they were working on already importing something, and then their new public import foo not working as they expect, and they'd have to end up knowing about the fact that the internal import is preferred to understand they need to go search for other uses and add public to them.

8 Likes
6

I like the direction you’ve outlined here.

There’s also another import-related feature I’d like to see, which Jordan touched on in the post you linked:

Suppose I write a library which vends a protocol. My library has no dependency requirements, but there are types in, say, Foundation and Numerics and Collections which could all conform to my protocol.

Currently, I have to make separate shim modules like “MyLibraryFoundationShim”, “MyLibraryNumericsShim”, and so forth, to provide the conformances, so that clients which don’t use those other libraries can still use mine.

But then clients of my library which do use those other libraries must remember to include the corresponding shims as well. This makes it a hassle for clients to use my library, since they need to import multiple dependencies instead of just one.

• • •

It would be much more ergonomic if my library could make a “convenience import” of those other libraries.

Then the conformances of, say, Numerics types to my protocol would be available for clients who import both my library and Numerics, but other clients who don’t import Numerics would still be able to use my library without any problems.

7 Likes
7

I like the public/internal distinction and agree that internal should be (or should have been) the default.

But it's been painful to watch the Java ecosystem struggle to migrate to the privacy of modules. One unmigrated library can foul many dependency trees.

In Swift, even applications may (should?) be composed of multiple modules, and need migrating. Applications targeting multiple OS platforms and versions might delay upgrading to a new language default of internal until all target platforms support it (a big lift if concurrency is not available on older platforms). This in turn stalls apps on old versions of libraries whose next version requires internal support. (Many Java libraries cite their clients when balking at converting to modules.)

I also like jordan rose on @_exported. Could this belong in package-level visibility control, where module/library publishers opt-in to internal?

E.g., let's say library L-1 in Package.swift can declare a dependency on a module L-0 to be internal, resulting in compiler errors when any L-0 types/names are used in the public API/ABI of L-1. That way, each library can opt in (or not) to using internal with their next version, and library clients manage around it at the same time they manage library version upgrades (i.e., they can remain on old versions if needed). And the new library itself can be used in older versions of Swift.

The scope change from per-import to per-module-dependency avoids the incongruity of multiple imports with different access levels and avoids the need for new access modifier on the import statement. But the cost is complexity: the full public API/ABI of module L-0 is visible to clients of L-1.

But that might be right. I'm a little uncomfortable with library/module L-1 selectively exposing types from module L-0 to application clients. That would mean there is a different view of L-0 for each L-n publishing L-0 types to app A (or for each import statement?). If the publisher of L-0 presents the only/fixed view of L-0, it might help incremental compilation and developer reasoning (at the cost of a larger API/ABI surface).

I'm not sure if this helps or hurts de-conflicting multiple internal dependencies on (different versions of) L-0. I'm prefer that happen early at package-declaration phase, rather than at link-time via discovery.

2 Likes
8

Wouldn't it be more in keeping with the current semantics (and the plans to support @_exported in the future) if

  • @_implementationOnly import translated to private import,
  • the meaning of import stayed the same as in Swift 5.5 (but potentially able to be also expressed as internal import); and
  • public import became a candidate for a potential, future equivalent to @_exported import)?
3 Likes
9

I like your proposal, it canonicalizes something that‘s already widely used with the private @_ wrappers.

10

Is that not served by the feature known as “cross-importing”?

6 Likes
11

I think the primary virtue of what’s proposed here is that it addresses the shortcoming that the current meaning of import isn’t internal to the module at all. For this reason I am quite fond of the plan as pitched by @beccadax. Any alternative that involves keeping the meaning of import as-is in today’s Swift maintains source compatibility but then also the principal problem that needs fixing.

2 Likes
12

By my reading of the pitch, this would be entirely a compile-time change and platform support would not be an issue. I may be misunderstanding though.

1 Like
13

I don't think so. This basically treats private, internal and public as meaning "lowest access", "default access", and "highest access", rather than as meaning "visible in this file", "visible in this module", and "visible in any module". It would be strange that you could use a type from a private import in an internal var, or one from an internal import in a public var.

If we do have something called private import, it should probably do what @Philippe_Hausler suggests upthread: only allow you to visibly use the type in private/fileprivate declarations, so the import is effectively an implementation detail encapsulated in that one file.

3 Likes
14

I see two sides to the question of integrating this with @_spi:

  1. The current @_spi(X) import M attribute adds to the import to allow access to the SPI group X, it doesn't affect the visibility of the import itself like the other import attributes discussed here. Because of this difference it could remain a distinct attribute that can be combined with the proposed public import and internal import. I don’t think we want to change its semantic but instead clarify it and maybe find a better attribute or syntax.
  2. What is more interesting is a new kind of visibility level for imports that we are considering to make a dependency visible at the same level as @_spi decls. This, let’s say, @_spiOnly public import M would allow the use of the APIs and SPIs of M in the local SPIs and be printed only in the private textual interface. All clients of the module would then load M iff they have access to the private interface, this would impose the same project or organization boundary from the current @_spi implementation. The syntax is still very much up for discussion and it could be better integrated with what the current pitch brings. I’d be interested to hear any ideas about this.
15

I'm a big fan of this, I often create files which encapsulate something I need to pull in from Foundation or another library (and expose an internal API instead). Having compiler support for this would be a +1.

Would it be possible to mix visibility of imports? For example:

public import struct Foundation.URL
import class Foundation.FileManager
6 Likes
16

This was a big pain point with using @_implementationOnly for me, so I'm really happy you want to fix this. :+1:

2 Likes
17

+1 to this pitch, although I'd love to see these exceptions (i.e. bugs) resolved.

I don't think clients should see extension, operators, etc for a module they do not directly import (unless the the module is explicitly exported like @Philippe_Hausler mentioned).

7 Likes
18

I quite like this proposal in general, so I’m +1, but I should note that this change does inflict most of its breakage on those who were not using the features we are adopting. That is, while it’s a strict quality of life improvement for those using @_implementationOnly imports, it will break almost all existing Swift frameworks today.

While @beccadax rightly calls out that these breakages should be easily fixed, I do just want to note that it does affect essentially all Swift code, even if they weren’t using the feature.

2 Likes
19

This sounds great! The way Swift’s import statements "leak" has always felt very unintuitive to me, and very inconsistent with the rest of the language design.

Like @George, I would also love the ability to use private import to import things as implementation details without leaking them to the whole module. Perhaps including that in the proposal would help make the change more palatable to framework authors who now have to change their code? "Yes you have to change all your imports, but you also get this new tool for scoping imports inside your module."

2 Likes
20

I love the direction this is going :clap::+1:! I have one question, however:

Why not use open import SomeKit to achieve the same result as the current @_exported import SomeKit? It feels to me like open being a more pervasive version of public when it comes to classes (namely, being public except with the added ability to subclass/override) would fit well with the imports, where (namely, open being public except with the added ability to re-expose the imported content as if it was defined locally). It even sounds right: "open import", as in "an import that is also open to the client code". Regarding the argument that current @_exported import occupies a very niche role and is not used too often, I can see a similarity with classes in this regard as well: Most classes are final because if their reference semantics (and all that it implies) and classes that are designed to be overridable (e.g. non-final) are much more rare, because the same effect can be achieved in most cases much better by using protocols (hence, the official guideline of marking all new classes as final unless an explicit need for subclassing arises). So, just like seeing open class is rare and indicates an unusual case, open import would indicate an unusual case for a very similar set of reasons.

1 Like