SE-0386: `package` access modifier

Evolution Proposal Reviews
,
102

Building on this: I gave some private feedback to the authors early in the proposal's development, and my first reaction was squarely in the "Why Don't We Just™ use @_spi for this?" camp. Specifically, I asked them something like this:

Consider this alternative design:

  • We turn @_spi(...) into a supported feature with some sensible name.
  • We introduce some kind of -default-spi-group <default-group-name> flag and have SwiftPM pass in the package name for it.
  • We treat @_spi with no group name as @_spi(<default-group-name>).
  • We add an implicit @_spi(<default-group-name>) to every import, suppressing warnings for modules that don't have any SPI with that name.

This alternative would cover more use cases. Why should we use your design instead?

And to be honest, I really thought that question would send them back to the drawing board. Instead, the authors successfully defended their design and convinced me that a new feature was the right solution.

They had a few good arguments, but the most important one was the library evolution differences. An important aspect of @_spi's design is that in a resilient library, SPI declarations have exactly the same ABI as public declarations; this allows you to build clients separately and mix-and-match library and client versions, and it also allows you to retroactively publish SPI from toasterOS 9 as API in the toasterOS 10 SDK. But adopting the behavior of public declarations places certain limitations on them, and it forces clients to generate slower "resilient" code that allows the library implementation to change.

We've long known that forcing all clients of ABI-stable libraries to generate resilient code was not ideal, and from the beginning we've discussed the notion of a "resilience domain", a set of modules which could access each other's guts non-resiliently, but which were therefore version-locked to one another and would have to be upgraded or downgraded as a unit. But this was always a theoretical concept, discussed in five-plus years of design documents but never surfaced as an actual language feature.

SE-0386 finally gives us a path to making this hypothetical feature into something you can actually use—and with such nice ergonomics that it can simply be the default in SwiftPM packages, and probably in other build systems as well. And, as a nice bonus, we also get a useful new access level.

To me, that seems like a powerful motivation to build something distinct from @_spi. It's not that @_spi is bad or that we shouldn't formalize it eventually—it's just that we can get much more by building a different feature to address this specific use case.

I am very sympathetic to this issue and I think we might not want to tie the keyword quite so closely to the concept of a "package", but other than a half-joking suggestion of friend, I don't have a great alternative to offer. Something based on team is probably better than package, but it's still a little misleading, since it makes perfect sense for a single engineering team to own several different "frogs". I'd prefer a word that says something about the modules, not the organization.

12 Likes
103

Honestly, I don't hate this.

Something I was having difficulty expressing was what you suggested: we should say something about the modules rather than how they're organized. More specifically, two or more modules passing -(package|frog)-name to the compiler indicate an agreement among them that they want to be able to see certain APIs that aren't otherwise visible to the outside world. I was struggling to think of a good term that described this idea of "two or more things mutually agreeing on something" as opposed to "the container inside which these things reside", and the concept of "friend modules" that share things with a "secret handshake" (the identifier passed via -(package|frog)-name) actually feels pretty natural.

And not tying us down to the overloaded term "package" frees us of the confusion that might ensue if we do want to make it possible for packages to opt-out or partition targets in some way other than the default.

3 Likes
104

how about subsystem?

1 Like
105

this is closest to my view, but i do not have an issue with the access modifier being tied to SPM, i have an issue with the access modifier being tied to versioning units.

i do not want to have multiple frogs within a frog namespace, or overlapping frog namespaces, i just want to have multiple non-overlapping frogs within the same versioning domain (SPM package).

to steal @allevato ’s idea, examples of such frogs might include:

let package:Package = .init(
  .executableTarget("foo-tool", frog: nil),
  .executableTarget("foo-example", frog: nil),

  .target("Foo", frog: "swift-foo"),
  .target("FooIO", frog: "swift-foo"),
  .target("FooParser", frog: "swift-foo"),

  .executableTarget("FooUnitTests", frog: "swift-foo"),
  .executableTarget("FooIntegrationTests", frog: nil),
  .target("FooAPITests", frog: nil),

  .target("FooBenchmarking", frog: "benchmarks"),
  .executableTarget("FooBenchmarkA", frog: "benchmarks"),
  .executableTarget("FooBenchmarkB", frog: "benchmarks")
)
1 Like
106

Right, I meant 1-1 with SwiftPM packages.

The benchmarks thing there is interesting. So you'd like to not just remove some of the targets from the standard frog for the package, but actually have multiple, non-overlapping frogs within the package? Can you talk about why that's important to you?

2 Likes
107

here is a (condensed) version of a manifest for a real library i maintain:

products: 
[
    .library(name: "BSON", targets: ["BSON"]),
    .library(name: "BSONDSL", targets: ["BSONDSL"]),
    .library(name: "BSONDecoding", targets: ["BSONDecoding"]),
    .library(name: "BSONEncoding", targets: ["BSONEncoding"]),
    .library(name: "BSONUnions", targets: ["BSONUnions"]),
    
    .library(name: "BSON_Durations", targets: ["BSON_Durations"]),
    .library(name: "BSON_OrderedCollections", targets: ["BSON_OrderedCollections"]),
    .library(name: "BSON_UUID", targets: ["BSON_UUID"]),

    .library(name: "Heartbeats", targets: ["Heartbeats"]),

    .library(name: "Mongo", targets: ["Mongo"]),
    .library(name: "MongoBuiltins", targets: ["MongoBuiltins"]),
    .library(name: "MongoDB", targets: ["MongoDB"]),
    .library(name: "MongoDSL", targets: ["MongoDSL"]),
    .library(name: "MongoChannel", targets: ["MongoChannel"]),
    .library(name: "MongoConnectionString", targets: ["MongoConnectionString"]),
    .library(name: "MongoDriver", targets: ["MongoDriver"]),
    .library(name: "MongoSchema", targets: ["MongoSchema"]),
    .library(name: "MongoWire", targets: ["MongoWire"]),

    .library(name: "SCRAM", targets: ["SCRAM"]),
    .library(name: "TraceableErrors", targets: ["TraceableErrors"]),
    .library(name: "UUID", targets: ["UUID"]),
],

based on the naming of the modules, hopefully we can observe that this package has (at least) two nucleii, a “BSON nucleus”, and a “Mongo nucleus”.

in my view, things that are closer to the BSON nucleus should be able to call certain things in other BSON-related modules that things close to the Mongo nucleus should not be able to call, and vice-versa.

you might ask: “why not just split up the package into two packages?” and the answer is that that would also decouple the versioning and make distribution all the more difficult. because SPM does a lot more things than just compile and link modules together, it also does dependency resolution, and forcing every package to have at most one nucleus means we can no longer use SPM’s dependency resolution capabilities to distribute the two nuclei together.

7 Likes
108

I think one‐to‐many makes sense. The package is the versioning node, pretty much by definition. In the past, every time we have tied anything else to it, it was never long before demand surfaced for the two to be decoupled. One example would be the system packages that later needed to be replaced by system modules. It is just my intuition, but I smell the same thing coming in the future here. So I think not tying it to the version any more than necessary would be wise. Due to the ABI considerations, stretching a frog beyond the boundary of the versioning node is a no‐go (fundamentally differing from SPIs). However, I see no technical reason to prohibit multiple frogs within the versioning node. Thus it naturally ends up one‐to‐many in my mind, i.e. one version to multiple (non‐overlapping, non‐nested) frogs. Although for ergonomics, one‐to‐one definitely makes a convenient implicit default.

As for the name of a frog, at the moment my gut says family, being commonly used metaphorically for a closely related group of things—and specifically close enough to share the same keys. Runners up are group, collection and bundle, but those smell either too general or too overloaded.

2 Likes
109

Currently, to access a symbol in another module, that symbol needs to be declared public . However, a symbol being public allows it to be accessed from any module at all, both within a package and from outside of a package, which is sometimes undesirable.

crossmodule ?

1 Like
110

I'll have a larger reply tomorrow, but my initial response is, why not just change the design of spi so that it doesn't require ABI exposure in all cases? Can't we just take the mechanics that have already been written for this proposal and change how it interacts with the language to use a new form of SPI declaration rather than an access level?

111

Because ABI stability is a relied-upon feature of @_spi(). If you remove it, you wind up back at this situation:

1 Like
112

That’s why I said “in all cases”. Of course we can continue providing the public visible version but it would be enhanced with other visibilities.

113

One fundamental issue I have with this proposal is that it does not completely solve anything:
Afaics it is about helping teams organising a codebase that has become too bulky — but there is nothing that stops a library from growing even bigger.
So, what are you going to do when package hits the same limitations it is supposed to cure? Add a new superpackage level?

4 Likes
114

I tend to agree with this, it seems a very natural principle when working on code organization.

But if I understand the proposal correctly, some of the assumptions at its foundation are related to the fact that, given that only you or only your team work on the codebase, you can set up practices and conventions that would prevent cross-nuclei calls.

This might be true, but it could still be the case the 2 different teams work on 2 different nuclei, so a consequence of the assumptions would be that the 2 nuclei should be grouped in 2 separate packages, but that would then cause the issues that you mentioned, related to distribution (as a side note, this affects large distributed libraries, but not apps development, where we can simply create many local packages to organize code, so this proposal would still be helpful for us without major drawbacks).

So, it seems that the issue reduces to 2 options:

  • either library distribution is made simpler and more convenient in case of 2 or more "internal packages" in a single library,
  • or a better use for the teamprivate access modifier would be to represent a group of modules within the same package.
115

This.

Why is the fundamental issue we are trying to solve here? As far as I can see, you have a module which defines an API and you want to be able to share it to some collection of other modules but not to the World in general. Does this proposal address that issue? Only partially. Sooner or later, somebody will come along and say "I have a package with modules A, B and C and I want B to see A.api() but I don't want C to be able to see it. The proposal will not work for that scenario nor will it work for any scenario where the modules cannot be divided into a number of disjoint sets for the purposes of sharing the non fully public APIs.

Also, I noticed that, conspicuous by its absence from the list of alternatives, was "do nothing". What do people do who want package private APIs at the moment? I guess you either don't document them or document them as private and unsupported. I understand that having a compiler enforced feature is far better, but the current state of affairs is not disastrous. It's not so bad that we need to implement an almost but not quite solution instead of taking the time to implement the right solution.

one more point on the subject of terminology: this feature superficially seems to elevate one particular distribution mechanism to being part of the language. I know the mechanics of how it works mean that it isn't so, but perception is king and this is how it will be perceived if we stick with package. I'm not comfortable with that.

4 Likes
116

I think fundamentally there are many different camps here. Some argue that we don't need many different types of access modifiers, but others (myself included) need more granular flexibility when it comes to expressiveness of visibility for certain API. Personally dropped using private and fileprivate altogether, just because they're either limiting or may not fit my own esthetic requirements. I also never use explicit internal unless there's possibly some ambiguity that would resolve it, nor do I design classes that can be subclasses. So my daily go to is the implicit internal and explicit public modifiers. I'm not trying to say that the other modifiers are useless, they're not. I just find that the whole access control is "not really modern" anymore. It's something that every new language keeps re-using because it seems to worked well here and there.

Let's face it, if we would remove the requirement to name access modifiers by old standards and for a second just consider a different approach, we will gain the flexibility everyone needs and improve upon those old school modifier.s

Yes naming is hard, but that's not the point here. Access modifiers as for now are not flexible enough to fit everyone's needs or styles. One could argue that this topic and decisions are opinionated, but does it really have to be that way?

Let's for a second remove everything from the access modifiers that adds non-visibility effects to the mix: subclassing, conformances, etc.

Just focus on raw visibility only. Here's what many people really want (names aren't the main priority of this example so accept them for the sake of this example):

scope
type
file
module
package

This list has no equivalent for public as I find it hard to come up with one. I mean, it could be external as it can be considered to be outside everything else (or world, all or whatever, just let's not name it public again for in this example).

If we now need to folder visibility for some good reasons, we can really easily introduce it. It would be additive and really non-brainer. I'm not referring to technical difficulties on how it can be implemented in the compiler, but purely on the reasoning of the existence and visibility order of such visibility modifiers.

I did exclude open for subclassing outside the module on purpose, because some current modifiers have their effects like subclassing inverted at certain boundaries. However, that's a discussion on certain effect defaults, rather than pure visibility of a type, its member or some global code entity.

So TLDR; from the viewpoint of pure visibility modifiers I'd be really happy to have a full set of flexibility modifiers that I could utilize.

scope
type
file
folder
module
package
external

I wonder if a major release like Swift 6 or even 7 could possibly fully reconsider this type of shift and guard it with the language mode or even just some other compiler flag. Today's modifier can be expressed by a combination of the above "visibility modifiers". The only difference, is that todays access modifiers have other effects merged into them as well, not just pure visibility.

private = type + file (+ implies module)
fileprivate = file (+ implies module)
internal = module
public = external 
open = external

NOTE: I'm not trying to say that everything is bad or something and I have no intention to sound silly or to get roasted just by providing my honest opinion on this part of the Swift language. So if you find something of this ridicules, please don't attack me in your response. I have zero intentions in a possibly toxic conversation, nor do I want to derail this proposal review. Thank you.

117

This is largely my opinion as well; we have "packages" here denoting a unit of code distribution that is contoured largely to coincide with SwiftPM packages, but we've already identified legitimate exceptions such that one can draw a Venn diagram that's not totally co-extensive.

It would seem to me that codifying a package visibility that is evocative of but doesn't have to be used exclusively with a SwiftPM package and with use cases we want to support that exclude parts of a package is fertile ground for confusion.

The idea of referring to a "team" seems nice; I also wonder if we couldn't lean into an adjective folks are already using in this thread and call this shared.

9 Likes
118

To talk about a concrete scenario where we'd like some sort of "package" visibility which I don't think this proposal handles well:

Realm's Objective C framework uses a custom modulemap file to define the modules Realm and Realm.Private. The Swift framework imports Realm.Private everywhere because whether or not something is part of the public obj-c API isn't relevant when working on the Swift framework. However, we have two different sets of obj-c tests. One imports Realm.Private and has unit tests for internal types and tests implementation details and such.

The other specifically imports only Realm and tests against the public API. We try to write as many tests in this way as possible, as we've found that it's both much more effective at finding bugs and it's occasionally helped us find API design problems. Quite often if a test can't be written using only the public API it's a sign that the public API is missing something which our users will need too. In addition, when tests use only the public API it means that having to change the test is a very strong sign that the change you're making will be a breaking change for the users. The fact that these tests can't accidentally use a private interface is thus a valuable features. We don't have to carefully check in code review that the author didn't make a mistake and can just trust the compiler to take care of ensuring that they stuck to the public API.

Swift currently doesn't have any way to do something similar. @testable is too coarse; it's designed to be enabled only in debug builds, but we have found it to be very important to also test our release builds. We don't want to ship release binaries with testing enabled due to that it makes all internals externally visible, so @testable doesn't let us test the thing we actually ship.

Package visibility (or whatever we want to call it) does do something similar to what we want. However, as noted it's valuable to have both tests which can view the special symbols and tests which can't. There's two general ideas for how to achieve this: decide at import time, or push the problem off onto the build system.

Package visibility chooses the latter, with the build system passing a --frog-name parameter that determines which symbols the thing being built has access to. I think that this is the wrong approach. Philosophically I think pushing things to the build system is nearly always a mistake. Given the choice between slightly more complicated code and a slightly more complicated build, I will pick the first 100% of the time. In the specific case of Swift, I think there's also major practical problems with pushing things off to the build system. For the purposes of this feature Swift has three build systems, as this'd need to be supported in all of SPM, Xcode and CocoaPods. Awkwardly, the third-party nearly unmaintained one is still the most popular option for importing libraries.

The other option is to do something at import time to indicate that you want something special. This is the choice taken by @_spi, @testable, and our custom modulemap approach, and I think it's the correct one. Having to type import(Private) RealmSwift (or whatever the syntax is) is less magical than relying on build system things and is not a meaningful burden. I honestly don't see what the downside of this is, and the entire package visibility feature appears to have been designed specifically to avoid this thing which to me is incredibly minor.

5 Likes
119

I'm not sure where to stand about the needs that need to be satisfied by this, though it seems package as designed is a bit too limited.

A design I think would be leaning to after reading this debate is something I'll call inhouse (in-house): an adjective meaning internal to an organisation, which isn't necessarily the same as a package. A module in the same "house" would have access to another module inhouse declarations.

We could have inhouse work like package in this proposal, except it'd be configurable in Package.swift in a way you can choose a different house for each module in the package. This way test modules can be put in their own house (or no house at all) and only have access to public APIs.

And perhaps as a future direction towards @_spi territory: we could allow houses to be either package-internal houses (limiting access to inside the package with more room for optimization) or public houses (accessible from outside the package). More design needed here.

I might have just given a name on ideas that have been floating here, but I feel having a believable name for a concept is important when evaluating and iterating on it. There's only so far you can go thinking about frogs before it becomes confusing.

2 Likes
120

i use a lot of __shared and having a non-underscored variant of __shared that does something completely different would be very confusing.

2 Likes
121

Underscored features are deliberately not meant to constrain options for Evolution, and I feel strongly that on principle we ought to pay this consideration no mind if we otherwise find a name to be well suited.

2 Likes