SE-0386: `package` access modifier

[DevAndArtist (Adrian Meister)]

I appreciate your feedback. Re "closed": While this looks fine on the spectrum of linear access modifiers, it's very much confusing in code. In fact public does already imply closed behavior in the context of disallowing to subclass from a public class. I wish this would have been done for protocols where a public protocol would mean that it's visible and you can use it algorithmically, but you cannot conform to it ("closed protocol"). In the flip side an open protocol would permit conformance to that protocol from outside. Unfortunately that ship sailed with the introduction of open.

---

All this just shows that the whole access modifier topic is is very inconvenient from many perspectives. I really wish we could finally overhaul it once and for all with the Swift 6 language mode and keep the old behavior only in previous language modes.

There's so much that this can enable in the context of library development, but right now we're just stuck in front of a dead end.

[DevAndArtist (Adrian Meister)]

Just to comment my thoughts on this for other review readers.

@farzadshbfn

I'm -1 for external as it is not mutually exclusive with public and open.

As I mentioned above, I don't think "external" is generally equals "public". Think about it from a different perspective. If you're working in a small team (that's your "module") inside a company (that's your "package") and you need to make your project available "externally". This does not imply that the project will be available outside of the company itself (that would be "public"). It's only made available externally for other teams to use (so it remains within the realm of a "package").

[jayton (Jens Ayton)]

@farzadshbfn

I would say "this function is package" is not the only way of addressing a function's accessibility.
I believe you could say:

• "This function has open accessibility"
• "This function has public accessibility"
• "This function has package accessibility"
  ...
• "This function has private accessibility"

One could say that, but how often does anybody actually do so? It’s not a coincidence that the existing accessibility modifiers are adjectives.

[Jeehut (Cihat Gündüz)]

I‘m +0.5 on “external”: It’s better than “package” and when compared to “internal” which is already scoped as “internal to this module”, it makes sense as it means “external to this module”, which is correct. But I’m not 100% into it, because even “external to this package” doesn’t specify where the accessibility ends, and that’s the most important aspect of an access modifier IMO.

Here’s another naming idea which is pretty close to the original name but also contains a similar notion to “internal”: inpackage.

Yes, “inpackage” isn’t a real word and it could be written as “inPackage” to be more clear about the word boundaries, but we already have “fileprivate” which also isn’t a word and it also doesn’t use camel casing.

What do you think?

[lassejansen]

How about public(package)?

public(package) var example: Int

public(package) private(set) var example: Int

public(package) internal(set) var example: Int

public public(package, set) var example: Int

This would fit with the suggestion from the future directions section to use open(package) to restrict subclassing to package modules.

 // Visible inside package, subclassable inside package
open(package) class Example {} 

// Visible outside of package, subclassable inside package
public open(package) class Example {}

[jeremyabannister (Jeremy Bannister)]

I see this:

Pre-Pitch: `@package` argument syntax

Hello folks at the Swift community! A year and a while ago, I started the work of supporting @package(...) import syntax for importing SwiftPM dependencies in Swift scripts as a GSoC project. During the project, I managed to work out a proof-of-concept of this feature with Swift Driver integration. The PoC implementation consists of a small standalone tool that depends on SwiftSyntax, since adding SwiftSyntax dependency to SwiftPM was a big trouble at that time. With the great work from the Sw…

as continued evidence that package is not the right name, even if adding a keyword for this exact access control setting is a good idea.

[DevAndArtist (Adrian Meister)]

@Jeehut

“external to this module”

There is a language out there which actually has external, but the semantics are different as in Solidity as far as I undestand it, it's forbidden to call that code internally.

External functions are part of the contract interface, which means they can be called from other contracts and via transactions. An external function f cannot be called internally (i.e. f()does not work, but this.f() works).

Not sure if Swift does ever need something like this.

---

@Jeehut

But I’m not 100% into it, because even “external to this package” doesn’t specify where the accessibility ends, and that’s the most important aspect of an access modifier IMO.

Well the ceiling for external is public. We don't know if the future evolution of Swift and SPM will introduce something that is again above a package and below what public offers, but right now it is and that's where external could exist.

[Jon_Shier]

• What is your evaluation of the proposal?

-1.

This is a limited, inflexible solution to a niche need that doesn't scale with the language. If this proposal covers a real need, there seems to be better, more flexible, and more scalable solutions than simply adding a new visibility modifier.

The proposal's casual dismissal of the existing @_spi features discards a legitimate and powerful direction to solve the asserted problem. Not only could the SPI feature be expanded to solve the issues identified in the proposal, but such an expansion would create a feature far more flexible and powerful than any expansion of access levels. Not only could it solve the package issue, but it could solve things like test-only API, or custom internal visibility for particular subsets of packages or targets.

• Is the problem being addressed significant enough to warrant a change to Swift?

Perhaps, though the motivation is rather theoretical right now. If Apple were to replace Xcode projects with package definitions, and not improve other aspects of packages at the same time, then there may be a need for something to customize visibility of certain symbols. Given the limited, inflexible, and unscalable nature of this proposal, it doesn't seem like a good idea to create a solution for a problem that doesn't really exist yet.

• Does this proposal fit well with the feel and direction of Swift?

Given the dismissal of pretty much all other access control pitches, most of which would have more immediate impact to the language and its users, I don't find this proposal matches with prior indicated direction.

• If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

No other language I've used had such visibility. Are there any that have an equivalent?

• How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

Participated in this review thread, read the proposal. Didn't participate in the pitch thread (IIRC) as I thought, like all visibility proposals, this wouldn't go anywhere.

[callek]

I'm +0 on the feature but I prefer packagewide, packageinternal or packageprivate to package.

[Jakub_Suder]

I'm a bit late, but I wanted to add one comment: I like the general idea, I think it will be a useful addition, but I'm -1 on the proposal as presented currently.

The problem I have with it is the choice to restrict subclassing within the package and outside of the module. This makes the new access level less useful, because it cannot be used for a subset of use cases (those where a parent class is defined somewhere within a package and then subclassed in other modules). If you want to build this kind of class hierarchy, your choices are either to keep all classes within one module, or rearchitect it differently, or make the parent class open, which of course makes it open to external users, which we wanted to avoid.

Note that we do not have the same problem here as in open vs. public: the modules within a package will be written & maintained by the same person/team. By allowing subclassing within the package, we don't create an API expectation that will have to be taken into account and maintained forever, as with marking a class open, because no one outside of the package will be using this API anyway. And the people who are working on the package can be expected to use it as designed, and can change the call site if needed. If a class needs to be exposed across a package that should not be subclassed outside of the module, it can be just marked as such in the documentation and caught in pull requests, or fixed later if necessary.

In other words: making package allow subclassing within the whole package creates a new feature in Swift that you can simply choose to not use if that's your chosen approach. Disallowing subclassing outside of the origin module would instead be adding a limitation to the new feature which cannot be disabled if someone chooses so.

So my preference would be to make package work as "Available in Package / Subclassable in Package".

[John_McCall]

Let me take my review-manager hat off for a moment and talk this through as a language designer.

Swift's central goal for access control is to encourage what we see as good program design, where you can break your code down into well-thought-out layers of libraries that can independently evolve and grow as you continue to work on them. This goal can sometimes come into conflict with the goal of feeling "lightweight", which in many ways comes down to discouraging programmers from brooding endlessly over details that don't really matter. That could certainly happen with access control, if it were too fine-grained.

The balance that Swift strikes is centered around the idea of co-development. Different groups of people work on different parts of a program. Swift's access control encourages you to define boundaries between parts that can be thought of as little library units: types, files, modules. That's how we promote that first goal of layering and code re-use. But as long as you have the same people working on the same code, these boundaries within the code aren't that important. People working on the same code need to be talking to each other anyway, and they can agree on standards, and they can repair any little breaches just as easily as they broke them in the first place. It's when the boundaries in code reflect real differences between groups of programmers and how they contribute their software to the whole that we first run into hard barriers to evolving the code; those are the boundaries that are truly important to enforce in access control. Without that in play, there's no reason we shouldn't keep things lightweight.

The problem we're looking at here is that the language doesn't let a group of programmers simultaneously enforce those boundaries with other groups — the only boundaries that are truly important to enforce — while also splitting their own code into multiple modules, unless the modules don't need to share any private interfaces with each other. That's a significant problem that I think is worth addressing.

The design of the @_spi attribute really is focused on needing to poke a very specific hole in access control across one of these boundaries between groups. That's why it requires a group name on both declaration and import, and it's why individual files need to import the SPI separately: those are precautions meant to strongly discourage other uses of the SPI. That might not be how some other people use the term "SPI" (although I wasn't aware this was a common term outside of Apple? The other uses I can find on the internet are unrelated), but it is explicitly how that attribute is designed. The effects of that are not what we want for boundaries that are purely internal to a group, because that gets us back to programmers brooding on unimportant things; even just in this thread, it sounds like the people who want to use the attribute this way are talking about drawing all sorts of fine-grained distinctions.

I am very open to the idea that package is not the right term for this concept of a bunch of code written by the same programming group. (Put aside other questions about the keyword for a second, like whether it should have a -private suffix.) The very nice thing about package as a term is that it directly invokes a very important use case for this: related library modules in a single open-source package. But that comes with two disadvantages that I can see:

• Other kinds of programming groups can have the same problem without necessarily thinking of themselves as writing a "package". The team writing the data layer for a medium-sized app might well have the exact same problem of wanting to split their code into multiple modules without giving access to their internal APIs to the UI layer, but they'd probably find it strange to see package throughout their codebase referring to their team's code.

• It's not necessarily true that everything in a SwiftPM package ought to have access to package-private APIs. I'm not particularly motivated by the idea of a package that contains code from teams that don't closely collaborate — that seems like a perfectly reasonable thing for SwiftPM to be opinionated about, and if that discourages using package in such packages, so be it. But @taylorswift's example of a sample project that should only be using public API seems much more compelling. I believe SwiftPM is looking at features for making this kind of supplemental code explicit in a package, so that might give an avenue for excluding them from the scope of package, but you can already see the difficulty here: it's hard to talk about this difference, and it might be a little surprising that anything in the package wouldn't have access to package-privatte APIs.

I wonder if it might not be better to use a keyword that recognizes the organizational differences being assumed here, something like teamprivate. Using a (slightly) longer compound keyword also has the benefit of calling out the unusually-expanded scope, the same way that fileprivate calls special attention to the fact that a declaration needs to be used from other scopes within the same file.

[Karl]

A lot of people like to talk about SOLID principles, and the "S" there stands for the Single Responsibility Principle. Sometimes you see that expressed as "A class should have only one reason to change", but I've noticed that formulation can be ambiguous. From the person who came up with the term:

this gets to the crux of the Single Responsibility Principle. This principle is about people.

When you write a software module, you want to make sure that when changes are requested, those changes can only originate from a single person, or rather, a single tightly coupled group of people representing a single narrowly defined business function. You want to isolate your modules from the complexities of the organization as a whole, and design your systems such that each module is responsible (responds to) the needs of just that one business function.

So it comes from the same sort of direction - a people-centric approach to encapsulation.

@John_McCall

The problem we're looking at here is that the language doesn't let a group of programmers simultaneously enforce those boundaries with other groups — the only boundaries that are truly important to enforce — while also splitting their own code into multiple modules

Right, and I think the reason people are attracted to SPI as a superior alternative is that it scales not just to multiple modules, but also to multiple packages. That's the thing I'm not really convinced about - is the package the maximum unit of programmer coordination?

I don't think it is. Sometimes I expose interfaces for benchmarking, or fuzz-testing, or to write private utilities, and I don't want any of those targets to be forced to live in the same package. They are not deliverables; they are team-internal tools which generally use the public API but, by their nature, should also be able to depend on implementation details.

Similarly, if I was organising something on the scale of Apple's SDK, I wouldn't want to make the entire SDK a single mammoth package including every single library and private utility - even though they are different modules which may need to share private implementation details.

I'd be okay with something based on the idea of team-oriented access, but it all depends on where those boundaries actually are. If I think about scaling this up to coordination across packages, I can't shake the idea that the solution will look almost exactly like @_spi. Maybe they'll be called "team names" or something, and maybe you won't need to declare them on every import.

[John_McCall]

@Karl

Similarly, if I was organising something on the scale of Apple's SDK, I wouldn't want to make the entire SDK a single mammoth package including every single library and private utility - even though they are different modules which may need to share private implementation details.

I’m not sure what you’re arguing here. Apple would never put the entire OS in the same “team” because that does not reflect the reality of how the OS is developed. We use @_spi for SPI at Apple — which is of course why the feature exists (unofficially) in Swift and has the design it does — precisely because changes to framework SPI need to be carefully coordinated and staged between different development teams.

I’ve never said that @_spi is a useless feature, just that its design is not what I think we want for intra-team interfaces. If you change the design by dropping two of the three distinctive aspects, then okay, but now we’re talking about a totally different design, basically teamprivate where a module can be part of multiple teams, and the obvious question is whether that flexibility really bears its weight.

If all you care about is letting different packages be part of the same “team”, I think the right way to do that would be to let that be configured in SwiftPM, not to muddy the concept of a team. I’ll leave it to other people to opine on whether that’s a good idea.

[ExFalsoQuodlibet (Elviro Rocca)]

@John_McCall

something like teamprivate

Let me start by saying that I completely agree with the gist of your post, as I understood it: optimizing for the "physical" boundaries of a codebase, including the boundaries between teams working on the same product, is more important than obsessing over specific visibility rules of specific members of specific types.

But the available accessibility identifiers will inevitably drive the structure of a codebase when it comes to code ownership. For example, because I can define something as fileprivate, I can make sure that the owner of that file, whether an individual or a team, will be informed and asked for permission if something changes there, and the owner can decide if some components declared in that file should be visible outside the file or not.

We can also take advantage of the concept of a "module" to enforce similar rules on a larger scale, so when a team owns a series of modules, it can use public to decide what should be seen outside the modules, by other teams, for example.

The current situation has limitations, though:

• there's no way to enforce visibility across the modules owned by a team, but not outside;
• once in a module, there's no way to further split it in smaller domains, for simple code organization even if the owner is still a single team, or for clarifying the boundaries between subteams

The package access modifier could help in this situation, but I think it will drive a specific way of using packages to distribute ownership among teams.

In our case, for example, there's a single app that's worked on by several teams. The app is modular, it contains some packages related to layers of the architecture, within which there are modules assigned to different teams, and we use folders to organize the modules per team.

What we're missing right now is a way to further split large modules into smaller components, without them exposing stuff to the rest of the app, but at the same time being able to share code across them.

This could be done in several ways. The feature introduced by this proposal would compel us to turn what are currently modules into packages, so we could have smaller modules inside. But the package, as a tool, seems to be not just for code organization within an app, considering all the attributes of the manifest and the package resolution step in Xcode, so the proliferation of packages in the app might not really match well with our needs.

A different idea could be that of a "submodule", that is, essentially, a logical boundary within a the module itself, that, in following the "physical boundaries matter" approach, could be realized with something like folderprivate, which I personally think would be a great idea.

So, I'm not opposed to the package modifier at all: I'm just saying that for large projects that are worked on by multiple teams, it might not be the right tool for the job, and could produce an unnecessary package proliferation just for team ownership needs.

[Karl]

@John_McCall

I’m not sure what you’re arguing here. Apple would never put the entire OS in the same “team” because that does not reflect the reality of how the OS is developed.

Right - that's exactly what I'm saying. There is generally not just one perspective of which "teams" exist.

To use a fictional, Apple-related example - let's imagine WebKit needs special APIs from audio or graphics subsystems. I'm supposing they are generally considered separate teams from the perspective of those within the organisation, yet because they are part of the same organisation ("Tim team Apple" if you like), they can collaborate and share private implementation details to a greater extent than they can with those outside of the organisation. They are part of multiple teams.

@John_McCall

If all you care about is letting different packages be part of the same “team”, I think the right way to do that would be to let that be configured in SwiftPM, not to muddy the concept of a team.

I think the concept of a team is muddy to begin with.

It seems to me that you are suggesting that team == package, and I do not agree with that. Again, consider whether WebKit and CoreAudio would need to be part of the same package in order to share a private interface were they both written in Swift. I think that shouldn't be required, and that collaboration can exist across packages to a similar extent as it can within a single package.

[jayton (Jens Ayton)]

@ExFalsoQuodlibet

The feature introduced by this proposal would compel us to turn what are currently modules into packages

Note that the feature as proposed doesn’t actually require “packages” for visibility purposes to correspond to SPM packages, or any other specific organization; the “package” name is specified by a command-line argument to the compiler, and in a non-SPM build system you should be able to use whatever unit of organization is appropriate for your use case as the package-name.

That said, I think that using the name “package” is inappropriate and misleading for exactly this reason.

[jeremyabannister (Jeremy Bannister)]

Thank you @John_McCall for taking the conversation in this direction - I was hoping someone with the knowledge and insight to do so would address the topic more holistically.

@John_McCall

something like teamprivate

I think I like the idea of teamprivate, but my understanding of exactly how that would work is a bit fuzzy. I look forward to reading more as the idea evolves.

I think you're questioning whether the package boundary would ever be a useful boundary to enforce, but I'm not totally sure, so I want to ask:

Do you think it is possible that we come to the conclusion that teamprivate, packageprivate and folderprivate all be added to the language because they all represent different boundaries that are worth enforcing in different situations? (The heart of the question being, do you see them as orthogonal features and do you see them each as potentially valuable?)

I know that there's a lot of dislike for fileprivate, which I don't currently know the foundation of, and I've said that I think it's a well-named and useful concept, and that I use it all the time. Without intending to spark any kind of detailed debate here, I just want to say that given my positive perspective toward fileprivate, I find the concept of packageprivate entirely unobjectionable and the concept of folderprivate quite exciting. The file, the folder and the package are all indisputable boundaries (and seem to me to be sibling concepts), one of which we already provide a modifier for that I make great use of, so to me filling in those gaps would be a big +1 (until/unless I'm convinced otherwise).

[jeremyabannister (Jeremy Bannister)]

@jeremyabannister

The file, the folder and the package are all indisputable boundaries

@jayton

Note that the feature as proposed doesn’t actually require “packages” for visibility purposes to correspond to SPM packages,

Ok, maybe "package" is disputed...

[ExFalsoQuodlibet (Elviro Rocca)]

@jayton

Note that the feature as proposed doesn’t actually require “packages” for visibility purposes to correspond to SPM packages, or any other specific organization; the “package” name is specified by a command-line argument to the compiler, and in a non-SPM build system you should be able to use whatever unit of organization is appropriate for your use case as the package-name.

I don't think this would apply to our case: in developing an iOS app, we're still pretty much in the SPM world, we have packages, targets et cetera and we would likely use this access modifier to mean exactly "within the current Swift package".

@jeremyabannister

I know that there's a lot of dislike for fileprivate, which I don't currently know the foundation of, and I've said that I think it's a well-named and useful concept, and that I use it all the time.

Same here. I get the impression that the criticism comes form a place where member access is pretty much tied to a object-oriented, class hierarchy-based world.

For example, to me it's completely natural, fine, maintainable et cetera to add an extension in a file – to a type not defined in the file – with fileprivate access, I do it all the time, I've been doing it for years, and I find that it conforms perfectly to the usual principles (SOLID, cohesion-over-coupling, you name it): one might even realize that without it you would need to follow some "design patterns" to compensate for the lack of it. This is also why I agree with John's point that we spend to much time focusing on things that don't matter that much.

In listing the undeniable physical boundaries I would say that

• package (current proposal),
• module (internal),
• folder (currently missing),
• file (fileprivate, or private at top level),

are the key ones, I would very much support a plan that fills the gaps.

[taylorswift]

@jayton

the “package” name is specified by a command-line argument to the compiler, and in a non-SPM build system you should be able to use whatever unit of organization is appropriate for your use case as the package-name.

i'm having a really hard time understanding how this is different from an SPI name. isn't this just assigning group membership to modules, but in a less granular way?