[Second review] SE-0386: `package` access modifier

Evolution Proposal Reviews
1

Hello, Swift community.

The review of SE-0386: package access modifier ran from January 26th through February 8th. There was a lot of feedback in the review, and some of the issues are subtle. The Language Workgroup spent a long time trying to decide what to do, which is why this announcement is a bit late.

Review feedback was mixed about the proposal. Many people wrote to say that they were very happy with the proposal; many people also expressed concerns about it, ranging from the design concepts to the surface syntax. One of the questions we always ask in a review is whether the problem being addressed is significant enough to warrant a change to Swift. The feedback was fairly clear on that, at least: there is definitely a real need for giving related modules access to declarations without going so far as to make them public. The discussion is whether this proposal is the best way to achieve that.

Specificity of control and the @_spi feature

Some community members felt that the proposal's approach was insufficiently general. package as proposed allows indiscriminate access within a single group of related modules. However, there are use cases where it's helpful to be more judicious about the users of an API, discouraging access outside of its intended clients. There's also fair reason to be concerned about allowing only a single level of grouping above the module level; maybe some people will want a hierarchy of such groups. Swift already has an unofficial feature that allows some cross-module access, @_spi, and most of these community members would like us to embrace and generalize that rather than just adding a new access level.

The Language Workgroup believes that adding a new access level is the right thing to do.

Imagine you're writing an API to which this feature might apply. You don't want to simply make it public, so you must have concerns about it being used by untrusted clients. One way you could protect it would be to enumerate all the clients you trust with it. To do that, you'd need a way to specify all of those clients, maybe even as narrowly as a specific function. You'd also have to update that list every time you found a reason for new code to use the API. That might be a little onerous sometimes, but it'd always be very clear which clients were allowed to use it.

In principle, a programming language's entire access control system could be designed that way: every declaration is either public, private, or has an exact list of the entities allowed to use it. No doubt, some people would find that design abstractly appealing. But it's interesting to note that Swift already doesn't take that approach within a module. If you want a declaration to be usable elsewhere in a file, you can make it fileprivate; if you want it to be usable elsewhere in a module, you can make it internal; but you can't be more precise than that. This is a deliberate decision, and it is based on an assumption about what it means to group some code into a module.

A lot of code is only used within what you might call a "development process boundary". That is, the code and all of its clients are part of the same "codebase", and they are developed, built, and distributed to clients in lockstep. It is easy in such an environment to reason about all the uses of something: IDEs (or in simple cases, a simple text search) can quickly find all the references to a declaration within a program. Maintaining explicit access lists as the code evolves would be a significant burden with little upside. Furthermore, barriers like this within a codebase are largely enforced socially anyway: if you can't trust your coworker (or Future You) to not use a declaration they're not supposed to, even if it has a scary comment on it saying it should only be used by the X subsystem, you also can't trust them not to just add their code to the access list or just make the declaration public.

Swift modules are always built in a single step, and so they must be deployed in a single step. Swift therefore assumes that it's reasonable to think of the code within a module as developed in lockstep --- even though, in theory, you could certainly have a single module built from hundreds of files submitted independently to the build by different developers with little cooperation. But Swift is an opinionated language, and we think that's not a good way to write a module, and so we decline to offer language tools that would be helpful to programmers in such a situation. Instead, we would encourage those programmers (assuming they can't just learn to cooperate) to divide their module into separate modules so that they can better encapsulate their submissions by taking advantage of internal access control.

A single "codebase" developed in lockstep can certainly include multiple modules. We want to encourage people to make small, tightly-abstracted modules, and sometimes it's easiest to do that if they can still share a few private interfaces. (Even if you intend to whittle them down over time, the ability to start out with private interfaces can make it a lot easier to split up a large module.) The same reasoning above about specificity still applies across modules in these cases, since the modules are still developed in lockstep. So the Language Workgroup feels that this proposal embraces and supports the same principles of good development that motivate our approach to intra-module access control.

That's not to say that it's never appropriate to be more specific. When a declaration does need to be exposed to clients outside of that development process boundary, being specific is highly desirable. The @_spi feature has been built around that idea; in fact, a major part of why we haven't made it an official feature yet is that we feel it isn't specific enough. The Language Workgroup is not closing off the possibility of making something like @_spi official, but we do not think it will ever be a good way to address the use pattern of closely related modules within a single codebase. Nor do we think there is a viable path to generalize @_spi to cover both use patterns; the resulting feature would be so complexly configurable that it would essentially become its own programming language, with attributes and rich interrelationships with modules.

Finally, while it is possible that a hierarchy of module groups would be useful to some programmers, we think that's unlikely. There is usually a clear boundary between the code that's developed in lockstep and the code that isn't. We are also conscious that access control is never perfectly precise, and as discussed above, we don't usually want it to be. It is better to have a simpler feature that pushes people towards good, common patterns than to have a more general feature that's slightly more precise in corner cases but is more awkward and painstaking to use in normal practice.

The package keyword and the nature of packages

The choice of package for the modifier gave rise to a lot of conceptual discussion about how to define a package and whether "package" is the right word at all. This can be thought of as two related questions. The first is whether "package" is a good general name for this concept of a group of modules. The second is whether the boundaries of SPM packages are the right place to be drawing the line, as opposed to allowing it to be drawn within or across packages. The Language Workgroup considered these questions very carefully; they were the primary focus of our deliberations.

"Package" is widely used in many different programming ecosystems as the name of the basic unit of code distribution. That is especially true for open source libraries, but many closed-source developers use the same mechanisms for managing their internal libraries. The boundaries of those units of code distribution almost always match our intuition above about what code is part of the same lockstep development process. There are exceptions, e.g. when a single package aggregates several loosely-coordinated projects, but they are very uncommon, and one could argue that they reflect poor package design. The Language Workgroup is loathe to pick another name that would be nearly synonymous with "package" in the vast majority of cases.

Not all Swift code is part of an SPM package, but not all Swift code needs to use this feature. The package modifier described in this proposal is useful whenever there's a group of library modules that wish to share a private interface between them without exposing that interface downstream to their clients. The Language Workgroup is comfortable saying that anyone in this situation should think of themselves as writing a package even if they aren't necessarily using SPM. The basic language mechanism is not tied to SPM; any build system can pass down the right compiler flag to organize a few modules into a package as far as the language is concerned.

A lot of reviewers were concerned about the way that the proposal's application to SPM packages makes package include every module in the manifest. There were several good arguments for why this was a problem. For example, a "black box" test should test what it would be like to use the package from outside the package, so it should not have special access to package declarations. Other reviewers noted that splitting an SPM package up into multiple packages was a relatively heavyweight step, and so they sometimes found it useful to have multiple logical groupings within a single manifest; they would like to be able to make those groups different "packages" for the purposes of the package modifier.

The Language Workgroup agrees that excluding targets from package is a critical missing capability and asked the authors to revise the proposal to include it. We also asked the authors to simultaneously consider the ability to divide packages into multiple groups. In response, the authors have added a group: argument to SPM targets which allows them to be excluded from the default package group. This argument could be extended to allow targets to be placed in named groups, but the authors have left that as a future direction because it's unclear whether this is the right way to express that structure to SPM.

Grammar of the package keyword

Many reviewers were concerned about using package as a modifier because it is neither an adjective nor immediately suggestive of access control. There were many suggested alternatives here, but the Language Workgroup did not find any of them satisfying. protected would be an attractive option except that it is already in widespread use with a specific, very different meaning. A compound keyword like packageprivate would be quite long, and the Workgroup felt that it didn't add much that users wouldn't already understand from package. The Workgroup was not concerned that users would confuse package for an introducer that declared a package.

Subclassing

Several reviewers expressed concern about the lack of support for classes that can be subclassed but only within the package. This is a common pattern in code that idiomatically relies on subclassing, such as when working with Apple's AppKit and UIKit frameworks. The Language Workgroup shares this concern, but we're comfortable with not addressing it in this first proposal; unfortunately, finding a good syntax for specifying the spectrum of possibilities here is difficult because of how the current open vs. public syntax conflates two dimensions of access control (use vs. subclassing/overriding). In the meantime, programmers in this situation will have to continue using one of their current (admittedly unappealing) options: either make the class open or put its subclasses in the same module.

@usableFromPackageInline

The Language Workgroup requested that this attribute be removed from the proposal. The attribute allows an internal declaration to be usable in package @inlinable code. Since clients with access to package declarations are already being trusted with special access to the module's internals, the Workgroup felt that there was not a good enough reason to allow expressing this exact possibility instead of just asking users to make the declaration package .

Package names

Initially, the proposal tightly constrainted the package name that was passed to the compiler. Because the package name is deliberately not exposed in the source language, there is no reason to do this, and the Language Workgroup has asked these restrictions to be lifted. The compiler will simply treat package names as Unicode strings and compare them for equality.

Second review

In keeping with these conclusions, SE-0386 has been accepted in large part. It has been returned for a second round of review, from today until April 10th. That review is limited to the areas of revision, all of which were discussed above but which can also be seen in this evolution PR.

Reviews are an important part of the Swift evolution process. All review feedback should be either on this forum thread or, if you would like to keep your feedback private, directly to the review manager by email. When contacting the review manager directly, please keep the proposal link at the top of the message and put "SE-0386" in the subject line.

What goes into a review?

The goal of the review process is to improve the proposal under review through constructive criticism and, eventually, determine the direction of Swift. When writing your review, here are some questions you might want to answer in your review:

  • What is your evaluation of the proposal?
  • Is the problem being addressed significant enough to warrant a change to Swift?
  • Does this proposal fit well with the feel and direction of Swift?
  • If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
  • How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

More information about the Swift evolution process is available at:

https://github.com/apple/swift-evolution/blob/main/process.md

Thank you,

John McCall
Review Manager

18 Likes
2

I sure feel like a lot of these kinds of issues could’ve been addressed by using Java-style reverse-domain name module naming.

3 Likes
3

my observation having participated extensively in the original review thread, was that median reception to the proposal was mixed to negative.

having watched the steady march of various “package keyword” PRs getting merged into apple/swift over the past month, i was quite concerned that the language workgroup was simply going to ignore all feedback and forge ahead anyway. but i am relieved that the workgroup is taking the need to partition modules within a package seriously, and that the proposal now includes a means for excluding individual targets from the global “package” group. this is a major step forward.

unfortunately, i think that simply excluding individual targets is not enough, we need to be able to add excluded targets to a different, named “package” group, otherwise this feature is effectively unavailable to the second or third, etc. nucleus within a versioned package distribution. and in the short term, this puts pressure on packages to split up into multiple subpackages, which is really really bad, because packages are like countries: once you split them up it’s almost impossible to put them back together again, since now you have stable versioning and module aliasing to think about.

this is already outlined as a “future direction” in the proposal text:

The group setting per target in Swift Package Manager allows a target to opt out of a package boundary, but if there is a need for creating multi-groups within a package, the setting could be expanded to allow that by introducing a .named(GROUP_ID) option, where GROUP_ID is a unique identifier for a specific group the target belongs to.

enum Group {
  case package
  case excluded
  case named(String)
}

For example, the new option could be specified in a target setting in a manifest like so:

  .target(name: "Game", dependencies: ["Engine"], group: .named("Core"))

i think this is a perfectly sensible design, that would completely fulfill the multinuclear use case, and should (must?) be incorporated into the proposal itself.

5 Likes
4

I appreciated very much reading such a thorough description of the topic and the workgroup's reasoning.

The following paragraph left me simultaneously with delight and with doubt:

Here are my thoughts:

  • Yes, totally, factoring out some code into a new package feels very heavyweight, and is a real pain point for me.

  • Therefore, on the one hand I'm very happy to have that taken into consideration. I've always read any mention of "submodules" that I come across on these forums very carefully.

  • On the other hand, it sounds like the term "package" is really considered to be the right fit to describe a group of connected "modules".

  • The reason I say "on the other hand" is because based on my understanding at the moment it seems to me that dividing Swift packages into "groups" is pretty much incompatible with using "package" as the official term for a group of connected modules. What I mean is:

  • Isn't it the case that a requirement of the final design is that "If I'm writing a Swift package then the package keyword will make my declaration accessible throughout the package"? But isn't the intention behind "named groups" within packages to allow further access control enforcement within a package? But the intention is not to introduce some new group access modifier, right? (The reason I say "pretty much" above is because of course adding a new access modifier would make it compatible.) A declaration in a named group that is marked package is either accessible only within the named group, or from any named group in the same package (in which case, which access modifier do we use for "private-to-this-group", or if none, what will named groups be useful for?).

  • If my reasoning in the previous bullet point holds up, then rather than it being unclear if it's correct to have named groups in a package it seems to me, as I said, definitively incompatible.

  • Which leads me to my first constructive bullet:

    Could we:

    1. Commit to "package" as the term for a group of modules,
    2. not consider ever allowing named groups inside of packages,
    3. and invest in ways to make it significantly lighter-weight to create packages?

  • Vague ideas of allowing multiple Package.swift-like files to coexist come to mind, but I think I really don't have a deep enough grasp of the nuances of versioned code distribution to know if the heavy-weight-ness is inherent.

This is an example of what makes me think there may be things about Swift packages that are inherently heavy-weight, but I would like to read some more concrete stuff about this.

  • Thoughts? Am I missing something key?
2 Likes
5

Could you explain how this proposal interacts with independent groups of targets? All the examples I could find on the thread describe groups that are completely separate which implies to me that they should not be importing each other, so they do not see each other's public or package symbols.

Where does that pressure come from? The symbols that can be marked with package after this proposal are public today by definition, so they are already visible. That implies to me that the pressure has to be constant pre- and post-proposal. What am I missing?

6

@NeoNacho can speak to this better than I can, but yes, my understanding is that one of the key questions here for SPM is whether we should do this by making named groups within a single manifest or by allowing a single package to contain multiple manifests.

7

This is the core of what I was getting at in my post - are the following three things not mutually incompatible?:

  1. The right name for the next level of grouping after "module" is "package".
  2. A manifest represents a group of modules.
  3. A "package" may contain multiple manifests.

#1 and #2 are already true, and so I'm confused by your statement that #3 is on the table (unless I'm wrong that these things are incompatible).

If we modified #1 to:

  1. "package" refers to a collection of modules and packages (i.e., allow nesting)

then I would see the three as being mutually compatible, but in the context of this proposed package keyword which doesn't seem to leave room for nesting, my confusion remains.

8

That's right.

Generally speaking, I think that any kind of grouping concept should do more than just interact with the package keyword, e.g. several people on the original review thread were talking about how easy it is today to accidentally import a module from the "wrong" set of modules. That is something I think a grouping feature in SwiftPM should be able to solve as well and therefore it makes more sense to tackle it separately from the introduction of package visibility.

1 Like
9

this is precisely the point, these groups should not see each others’ package symbols, they should only ever see package symbols from modules in the same group as them.

the “heavyweightness” of packages is not relevant to me, this has way more to do with versioning (both semantic and non-semantic) than how difficult it is to split up a Package.swift manifest.

right now, a package has exactly one “version series” associated with it. when you split a package in two, suddenly you have two independently incrementing “versions”. hopefully, the problems with having multiple independently incrementing versions of the same logical package should be self-evident.

10

I think I discussed this in my initial post. The Workgroup believes that packages that contain multiple independent groupings of modules are very uncommon, and we would rather use an evocative term that matches common usage than fall back on some novel but vague word that would be unassailable because it doesn’t really mean anything. If that leads to occasional circumlocutions like “two logical packages in the same SPM package”, we’re okay with that.

11

the company i currently work at (a multibillion dollar household name) checks in all of its code into a single unified repository which contains tens of thousands of targets. we have a single package with a unified CI pipeline and a single logical sequence of “versions”, and many hundreds of teams that own sectors of that package.

i don’t think this is uncommon at all. perhaps Apple is just different.

i am 100% okay with this as well. just please do not assume everyone is developing in the “multiple parallel and sometimes conflicting version timelines” workflow that is so common among open source swift packages on GitHub, because… we aren’t…

1 Like
12

Ok, I think I've understood that this phrasing of mine is ambiguous, and also that I was wrong:

The unambiguous phrasing is:

"If I'm writing an SPM package then the package keyword will make my declaration accessible throughout the SPM package."

but this is wrong.

It will happen to be true at first, but if an SPM package is ever allowed to contain multiple "logical packages" then it will stop being true, because package will instead make the declaration private to the nested "logical package".

IIUC the Workgroup believes that if multiple logical packages are ever supported within a single SPM package then it will be a feature that is used infrequently enough that it’s ok that those users will have to always keep in mind that a declaration with the word package on it is private to the “group”, not the SPM package.

For the record on 'heavyweightness".

For the record, I think we're referring to the same thing. At least, I'm not referring to the difficulty of manipulating the text of a manifest file, quite the opposite (although Xcode on my M1 MacBook does literally hang for over a minute sometimes if a make a change to a particularly complicated manifest file that I have, which certainly adds friction to manipulating the manifest too, but I'm sure they'll fix that...). I'm referring to the process that I go through of making the repo and pushing the code and uploading a version tag, etc. when all I really want is to be able to separate out the manifest file for a subsystem of interconnected modules, so that it can remain untouched as I flesh out the manifest files of other subsystems. Not being able to insulate (i.e., move to a different file) more-stable module constellations from other actively evolving ones would, I believe, continue to feel like a consistent pain point for me even when the incredible performance problems I'm experiencing with Xcode + long manifest files are resolved.

1 Like
13

yes! aligning package nuclei with manifest files makes a lot of sense to me, i simply have a hard time envisioning SPM packages ever supporting more than one manifest per repository because SPM has been a one-manifest-per-package build system since its inception.

in my workplace, we refer to package nuclei as “projects”. each team owns a handful of "projects", but all of the "projects" are part of the same "package".

in the open source swift world, we sometimes conflate packages with projects, since so many open source "projects" are responsible for one "package" that we start thinking of them as the same thing. (i have also experimented with multiple packages per "project" in the past, which i found to be an astoundingly unproductive workflow and would not recommend in the slightest.)

perhaps it would be helpful if we spoke of “packages” and “projects”, instead of “SPM packages” and “logical packages”.

in my mind, a “package” is a distributional concept, and a “project” is an access level. maybe if we named the keyword project, the difference between packages and package nuclei would become clearer?

3 Likes
14

This is an inversion of the meaning of “project” and “package” in terms of an Xcode project that builds one or more targets that have a number of internal and/or external swift packages as dependencies.

15

We don’t intend to revisit the naming decision. This second review is just about the areas of revision. I’ve been engaging with that discussion in order to help people understand our thinking, but the matter is closed.

2 Likes
16

Package names

Initially, the proposal tightly constrained the package name that was passed to the compiler. Because the package name is deliberately not exposed in the source language, there is no reason to do this, and the Language Workgroup has asked these restrictions to be lifted. The compiler will simply treat package names as Unicode strings and compare them for equality

That may only be considering the package name from the perspective of the compiler.

A lot of up- and down-stream build/assembly process could & would be built on naming conventions, but could depend on a distinction between simple and composite names for namespaces, if not versioning/platform conventions.

package on the supply/declare side reflects internal coherence and doesn't care about the name. But on the demand/use side there's a wide variety of composition relations from the external perspective besides simply using a library: peeking, vending, adapting, tweaking, supplying delegates, coordinating versions, platform adaptation, etc. These are the province of the build systems/SPM, test drivers, binary repositories and caches, uber-linkers, platform package managers, etc. What would make their work easier?

And for users, even if every name is actually just one namespace (like Java packages, where containment confers no privileges), semantically it makes a lot of sense to users to have containment, and to distinguish parts of a name like a version.

Dot- or slash-delimited identifiers are the minimalist form. URL's have more structure but can be surprisingly complicated. Wouldn't it be easier to start with the minimal form and grow on demand? Otherwise if names are unconstrained, any name-based conventions become costly to coordinate (or prone to contention), and a lot of known and unknowable futures would be foreclosed.

17

I think Boris question is (paraphrasing):

Does these different target groups actually import each other? If they don't, they won't see any symbols anyway? Marking something as package won't auto-import it into all other targets.

18

I'm glad that the proposal is moving forward, and I like the addition of the option to customize the package boundary, which opens the possibility in the future to create custom boundaries.

So, if I understand the future direction idea correctly, assuming that a manifest has 2 targets declared like this

.target(name: "Foo", group: .named("FooBar"))
.target(name: "Bar", group: .named("FooBar"))

if in the module Foo there's a declaration like this

package let visibleInFooAndBar = 42

the visibleInFooAndBar constant will actually be visible in both the Foo and Bar modules, but not in other modules in the same package, and not outside the package. Is that correct? If it is, I think it's a very exciting future direction, that will definitely help me and my team in our use cases.

So, I definitely support the proposal. But I disagree with this:

The need for multiple independent groupings is definitely the case in my experience, and I don't see how this is not going to be an issue for any sufficiently large project that's distributed as a single unit.

4 Likes
19

Agree definitely. It’s a direction we’re moving in too to try to reduce the number of packages we manage.

3 Likes
20

At least half of these categories won’t be affected at all. For the rest, what would make their work easier, in the aggregate, is for the compiler not to presuppose any convention and instead let any such enforcement happen at another level.