i'm having a really hard time understanding how this is different from an SPI name. isn't this just assigning group membership to modules, but in a less granular way?
1 Like
I’m unclear on what the functional difference is, except that with @_spi you need to annotate imports and with package you don’t.
You currently need to annotate imports. Given the effort put into this proposal instead focused on the SPI feature (or whatever we want to call it), we can make it look like whatever we want. Heck, it doesn't even need to be called SPI at the end of the day. It's the capabilities we want, and it provides a convenient focus point when discussing alternatives to this proposal.
2 Likes
Well, let's talk through this. Suppose we clone the @_spi attribute — I don't want to argue about names, so let's call it @frog — and remove the need to annotate imports. So you can write @frog("Private") on a declaration, and clients can use that declaration without needing to do anything. If that's all we do, the attribute is non-functional as access control; the declaration might as well just be public. So we need some way to say that some clients can't access the declaration, or better, for clients to opt in to being able to access it. The most logical place for that in source code is an annotation on the import, but here we specifically want to remove that, so the only reasonable alternative I can see is to make it a compiler flag, something like -enable-frog Private.
Now, that specific spelling is a bit problematic, because Private is apparently global. For one, that means the compiler doesn't know which modules are supposed to be providing MyName frogs, which I guess means it just has to expose any frogs it sees with that name for any module being imported. If you specifically needed the MyName frog from a particular module (which seems likely), and your build environment is misconfigured so that that's not available, your code will not build correctly, and there won't be any immediate clue as to what went wrong. Even more importantly, though, it means there's a real risk of collision, so it's actually important that we not call frogs something like Private, or else clients will be pulling in the Private frogs from every single module they import.
To solve the problem with collisions, we need frog names to be namespaced. To solve the build-system problem, we need frog namespaces to be tied in some straightforward way to modules so that we can know for a particular namespace which modules to expect that frog for. The logical conclusion is that frogs should be namespaced either by module or by some sort of meta-module. I think any sensible language design that drops import annotations is going to have to follow this line of reasoning, at least to this point.
Now, having come that far, this proposal is suggesting a very simple system. Once you've identified a grouping of modules to serve as a meta-module, the proposal says, you don't really need separate frogs within that grouping, and you don't really want any of the frogs in that grouping to be usable outside of that grouping. That simplifies the language design quite a bit, because you don't need a frog name anymore, so you can just write @frog; and @frog is effectively now just an access modifier that's somewhere between internal and public.
The argument for why those restrictions make sense is basically what I said above: once you've removed the requirement of import annotations, you have a kind of access control that's very lightweight and permissive about use sites within the scope of who can access it. Given that you do want to restrict uses — since you didn't just make the declaration public — you must be relying to some degree on programmer discipline to ensure that the declaration is used appropriately. That's a perfectly reasonable choice, as long as the scope of who can access it is stays within a group of programmers with relatively good internal communication, code review practices, low barriers to fixing inappropriate uses, etc. As soon as that stops being true, using this kind of permissive access control stops making much sense, and you really should be giving it the full @_spi treatment (assuming you aren't willing/ready to commit to a public API).
Most of the broad counter-arguments I'm seeing here fall into one of three camps:
-
People who really do want a feature with the exact design of
@_spi. This is a fine thing to want, but I don't think it's relevant as an objection to this proposal, for the reasons I've already stated:@_spiaddresses a related but crucially different use case that requires a different design approach. -
People who want to have multiple frogs within a frog namespace or different but overlapping frog namespaces. Right now, this seems very unmotivated to me. In my experience, even in very complex development situations, there's usually a pretty clear line that you can draw between inside and outside of your development group. If you're talking about internal APIs within your development group, it's best to stick with loose, non-specific access control that you maintain with discipline, communication, and code review. If you're talking about poking a targeted hole for the use of a client that's outside your development group, you really don't want that, and you need something like
@_spi. To take this objection more seriously, I need to understand what concrete problems people think that would solve with this. -
People who are fine with most of the basic design of this proposal but don't like that the access modifier is tied directly to SwiftPM packages. This is something I'm very interested in getting more discussion.
I don't mean that to be an exhaustive list of the feedback in this thread; there's a lot of good, specific feedback like Jakub's point about subclassing above. But these seem to be the gists of the broad disagreements with the design that I see.
12 Likes
I really like this framing because it grounds the discussion in how humans get work done together. For me, though, a framing that emphasizes the team as a unit of access control misses a key aspect of how humans do software development. We need to care about the practical constraints of software development practices, particularly how libraries are built and distributed.
Imagine that you belong to a team that is responsible for several packages, some of which are interrelated. It would be fairly typical for those separate packages to be located in distinct git repositories, have independent continuous integration systems, and in some cases produce binary artifacts that are separately distributed. Some of these packages might have dependencies on one another, including dependencies on the internal implementation details of modules in the other packages. In this scenario, the impact of changing an implementation detail of a module is very different if the dependencies on that detail come from outside of or inside of the same package. If the dependents are inside the package, then the developer making the change can confidently address the effects of that change atomically in a single pull request. CI will build and test all the affected modules together and if there are binary artifacts then a new, self consistent bundle of those artifacts can be published. If, on the other hand, the dependents belong to another package then the developer must take that into account even if that other package is owned by the same team. The developer needs to identify the affected packages, decide how exactly to handle coordination of the breaking change, and then execute a much different workflow. Maybe they choose to open simultaneous pull requests in the repositories for each package, counting on the fact that their team's CI and distribution pipelines are set up to handle tied changes like this. Or maybe they'll think better of it and decide to stage the change because a breaking change between packages could result in temporary ABI incompatibilities for teammates who haven't rebuilt upstream packages and won't download new binary artifacts for a while.
The point is that if you assume a package is something that generally evolves independently of other packages (regardless of ownership), then the language ought to help you reason about which changes might impact independent evolution. The proposal takes an opinionated stance that a package is the basic unit that encapsulates modules that evolve together closely. I understand that isn't the case for every team and every development style, but I think it is a reasonable assumption and helps us keep the design focused on solving a particular problem well.
There are some objections you might have to my arguments above if you make different assumptions about how software is built and distributed. If your team's packages tend to always be built together from source, using SPM to model dependencies and coordinate changes by pinning versions, then sure, making simultaneous pull requests to separate repositories might be fine for you. Similarly, if you work on a constellation of packages that are compiled separately but all reside in a single repository, then yes, coordinating these inter-package changes is also less of a concern for you. However, Swift's design is heavily influenced by the requirements of developing software runtimes where the components are both compiled and distributed separately. Library evolution has an enormous impact on the design of the language. This proposal is unapologetically designed to be especially well suited to the development workflows where library evolution is a chief concern. I think we maybe haven't spelled that out as explicitly as we ought to in the proposal.
Finally, you might still be objecting to the proposal because it doesn't attempt to solve the problem of privileged inter-package coordination (@_spi) and therefore the story feels too incomplete. I agree that a feature like @_spi is very important and something we should be working towards also making a formal part of the language - this proposal is not meant to compete with that. But as John detailed, we developed the existing unofficial @_spi feature with different design constraints and I think those differing characteristics are fundamental enough to justify separate features rather than one super flexible but complicated feature. The team that put this proposal together has extensive experience with @_spi because we built it and we work closely with the large number of teams who use it at Apple to manage complexity in the SDK. Today, teams at Apple are using @_spi for both inter-package and intra-package coordination and we know from experience that the feature is clunky for the latter purpose. There is a lot of interest from users in a lower friction, cleaner mechanism for keeping package-private interfaces truly private.
I do think there are some aspects of the proposed package design that are unsatisfying. Within Apple, this access level would be clearer if it were named project because that's what a collection of modules that build together are more commonly called. But I think SPM packages are a fundamental part of the Swift ecosystem that we should bias towards using the package terminology. And @taylorswift's point about documentation and example targets in a package not fitting well into this story is a good one, but I don't think it's a fatal flaw. Having a mechanism to exclude targets from package access seems like a good solution.
Overall, my feeling even after taking all of the arguments that have been made into account is that we have actually struck the right balance in choosing packages as the unit for this access control level. It is opinionated and by itself it doesn't solve all the problems that individual teams have in managing complexity, but it is a coherent addition to the language that solves a problem that isn't addressed at all today.
5 Likes
This is essentially where I stand, and raised a similar concern about @package(...) imports for SwiftPM scripting: Pre-Pitch: `@package` argument syntax. It feels like a layering violation to use "package" in this sense in the frontend.
Now, we could hand-wave it/rationalize it by saying that we're defining a new concept in the frontend for a group of related modules and we're calling it a "package", and SwiftPM will treat all the targets in the same SPM package as being in the same "package access scope" and other build systems can define a package however they want. That feels close to sufficient.
But, then we have to weigh @taylorswift's concern that you emphasized about whether targets like examples should be in the same "package access scope". I think this is a valid concern and more likely than it seems. Take swift-format for example; we might benefit from a "package access scope" among our internal modules, but we also have an explicitly defined public API surface. The swift-format executable target should only be able to use the public APIs from the SwiftFormat and SwiftFormatConfiguration modules; it should not have access to any non-public APIs and I wouldn't want to accidentally use one by having it subsumed into the same access scope.
So, SPM would need a way to semantically delineate certain kinds of targets. Could we say that executable targets don't belong in the same package access scope? I'm not sure; answering that "no" satisfies the swift-format case, but there may be other use cases where a tool would want to use APIs from the same package but not make them public. The same goes for testing support targets; maybe you want something that can be imported by all your test targets in the same package but isn't public to external clients of your package or to other non-test targets in the same package.
Then, if we instead need a way to let certain targets opt-out of this kind of access, the once simple analogy of "anything in the same SPM package can use package decls" falls apart and it becomes a lot harder to explain to users and to reason about.
I think the reason folks are reaching for @_spi as a comparison is because it feels like what this proposal does is create an implicit, automatic, unutterable @_spi among a group of modules, but we're treating it as a separate concept (and I think the implementation uses a different concept instead of building on SPI). Does it make sense to align these in some fashion? Let's ignore the name "package" for the time being and steal John's "frog" instead. If we said that -frog-name did two things:
- Treat decls like
frog classas if they were declared@_spi(«generated frog SPI name») public class - Treat
import Xas@_spi(«generated frog SPI name») import XifXwas compiled with the same-frog-nameas the importing module
Does that reveal any possible improvements/directions?
If we avoid the package name, maybe we could have SPM treat all targets in the same package as having the same -frog-name by default, but also provide a way for targets in the package manifest to provide a custom frogName: instead? And if any target in the manifest provides a custom frogName: then the default behavior goes away and you have to be explicit throughout. So swift-format could look something like this:
let package = Package(
.executableTarget("swift-format"),
.target("SwiftFormat", frogName: "SwiftFormat"),
.target("SwiftFormatConfiguration", frogName: "SwiftFormat"),
.target("SwiftFormatRules", frogName: "SwiftFormat"),
.target("SwiftFormatPrettyPrint", frogName: "SwiftFormat")
)
That might strike a balance where we make the easy thing easy and obvious while still making the more interesting cases possible. And folks would still have explicit @_spi for those more fine-grained cases where the access boundaries aren't clean partitions around groups of modules.
But, we would still need to figure out what to call this frog; under a design like the above, I don't think package is suitable.
4 Likes
It sounds like a solution to this is to allow a SwiftPM configuration to define "packages", rather than introduce an actual package access control keyword (which has many downsides noted in this thread).
People who want to have multiple frogs within a frog namespace or different but overlapping frog namespaces. Right now, this seems very unmotivated to me.
I'll note that this is the same crux of saying modules in packages need a special keyword rather than just having access to internal. I find this special layer of granularity within a package also quite unmotivating.
Building on this: I gave some private feedback to the authors early in the proposal's development, and my first reaction was squarely in the "Why Don't We Just™ use @_spi for this?" camp. Specifically, I asked them something like this:
Consider this alternative design:
- We turn
@_spi(...)into a supported feature with some sensible name.- We introduce some kind of
-default-spi-group <default-group-name>flag and have SwiftPM pass in the package name for it.- We treat
@_spiwith no group name as@_spi(<default-group-name>).- We add an implicit
@_spi(<default-group-name>)to everyimport, suppressing warnings for modules that don't have any SPI with that name.This alternative would cover more use cases. Why should we use your design instead?
And to be honest, I really thought that question would send them back to the drawing board. Instead, the authors successfully defended their design and convinced me that a new feature was the right solution.
They had a few good arguments, but the most important one was the library evolution differences. An important aspect of @_spi's design is that in a resilient library, SPI declarations have exactly the same ABI as public declarations; this allows you to build clients separately and mix-and-match library and client versions, and it also allows you to retroactively publish SPI from toasterOS 9 as API in the toasterOS 10 SDK. But adopting the behavior of public declarations places certain limitations on them, and it forces clients to generate slower "resilient" code that allows the library implementation to change.
We've long known that forcing all clients of ABI-stable libraries to generate resilient code was not ideal, and from the beginning we've discussed the notion of a "resilience domain", a set of modules which could access each other's guts non-resiliently, but which were therefore version-locked to one another and would have to be upgraded or downgraded as a unit. But this was always a theoretical concept, discussed in five-plus years of design documents but never surfaced as an actual language feature.
SE-0386 finally gives us a path to making this hypothetical feature into something you can actually use—and with such nice ergonomics that it can simply be the default in SwiftPM packages, and probably in other build systems as well. And, as a nice bonus, we also get a useful new access level.
To me, that seems like a powerful motivation to build something distinct from @_spi. It's not that @_spi is bad or that we shouldn't formalize it eventually—it's just that we can get much more by building a different feature to address this specific use case.
I am very sympathetic to this issue and I think we might not want to tie the keyword quite so closely to the concept of a "package", but other than a half-joking suggestion of friend, I don't have a great alternative to offer. Something based on team is probably better than package, but it's still a little misleading, since it makes perfect sense for a single engineering team to own several different "frogs". I'd prefer a word that says something about the modules, not the organization.
12 Likes
Honestly, I don't hate this.
Something I was having difficulty expressing was what you suggested: we should say something about the modules rather than how they're organized. More specifically, two or more modules passing -(package|frog)-name to the compiler indicate an agreement among them that they want to be able to see certain APIs that aren't otherwise visible to the outside world. I was struggling to think of a good term that described this idea of "two or more things mutually agreeing on something" as opposed to "the container inside which these things reside", and the concept of "friend modules" that share things with a "secret handshake" (the identifier passed via -(package|frog)-name) actually feels pretty natural.
And not tying us down to the overloaded term "package" frees us of the confusion that might ensue if we do want to make it possible for packages to opt-out or partition targets in some way other than the default.
3 Likes
how about subsystem?
1 Like
this is closest to my view, but i do not have an issue with the access modifier being tied to SPM, i have an issue with the access modifier being tied to versioning units.
i do not want to have multiple frogs within a frog namespace, or overlapping frog namespaces, i just want to have multiple non-overlapping frogs within the same versioning domain (SPM package).
to steal @allevato ’s idea, examples of such frogs might include:
let package:Package = .init(
.executableTarget("foo-tool", frog: nil),
.executableTarget("foo-example", frog: nil),
.target("Foo", frog: "swift-foo"),
.target("FooIO", frog: "swift-foo"),
.target("FooParser", frog: "swift-foo"),
.executableTarget("FooUnitTests", frog: "swift-foo"),
.executableTarget("FooIntegrationTests", frog: nil),
.target("FooAPITests", frog: nil),
.target("FooBenchmarking", frog: "benchmarks"),
.executableTarget("FooBenchmarkA", frog: "benchmarks"),
.executableTarget("FooBenchmarkB", frog: "benchmarks")
)
1 Like
Right, I meant 1-1 with SwiftPM packages.
The benchmarks thing there is interesting. So you'd like to not just remove some of the targets from the standard frog for the package, but actually have multiple, non-overlapping frogs within the package? Can you talk about why that's important to you?
2 Likes
here is a (condensed) version of a manifest for a real library i maintain:
products:
[
.library(name: "BSON", targets: ["BSON"]),
.library(name: "BSONDSL", targets: ["BSONDSL"]),
.library(name: "BSONDecoding", targets: ["BSONDecoding"]),
.library(name: "BSONEncoding", targets: ["BSONEncoding"]),
.library(name: "BSONUnions", targets: ["BSONUnions"]),
.library(name: "BSON_Durations", targets: ["BSON_Durations"]),
.library(name: "BSON_OrderedCollections", targets: ["BSON_OrderedCollections"]),
.library(name: "BSON_UUID", targets: ["BSON_UUID"]),
.library(name: "Heartbeats", targets: ["Heartbeats"]),
.library(name: "Mongo", targets: ["Mongo"]),
.library(name: "MongoBuiltins", targets: ["MongoBuiltins"]),
.library(name: "MongoDB", targets: ["MongoDB"]),
.library(name: "MongoDSL", targets: ["MongoDSL"]),
.library(name: "MongoChannel", targets: ["MongoChannel"]),
.library(name: "MongoConnectionString", targets: ["MongoConnectionString"]),
.library(name: "MongoDriver", targets: ["MongoDriver"]),
.library(name: "MongoSchema", targets: ["MongoSchema"]),
.library(name: "MongoWire", targets: ["MongoWire"]),
.library(name: "SCRAM", targets: ["SCRAM"]),
.library(name: "TraceableErrors", targets: ["TraceableErrors"]),
.library(name: "UUID", targets: ["UUID"]),
],
based on the naming of the modules, hopefully we can observe that this package has (at least) two nucleii, a “BSON nucleus”, and a “Mongo nucleus”.
in my view, things that are closer to the BSON nucleus should be able to call certain things in other BSON-related modules that things close to the Mongo nucleus should not be able to call, and vice-versa.
you might ask: “why not just split up the package into two packages?” and the answer is that that would also decouple the versioning and make distribution all the more difficult. because SPM does a lot more things than just compile and link modules together, it also does dependency resolution, and forcing every package to have at most one nucleus means we can no longer use SPM’s dependency resolution capabilities to distribute the two nuclei together.
7 Likes
I think one‐to‐many makes sense. The package is the versioning node, pretty much by definition. In the past, every time we have tied anything else to it, it was never long before demand surfaced for the two to be decoupled. One example would be the system packages that later needed to be replaced by system modules. It is just my intuition, but I smell the same thing coming in the future here. So I think not tying it to the version any more than necessary would be wise. Due to the ABI considerations, stretching a frog beyond the boundary of the versioning node is a no‐go (fundamentally differing from SPIs). However, I see no technical reason to prohibit multiple frogs within the versioning node. Thus it naturally ends up one‐to‐many in my mind, i.e. one version to multiple (non‐overlapping, non‐nested) frogs. Although for ergonomics, one‐to‐one definitely makes a convenient implicit default.
As for the name of a frog, at the moment my gut says family, being commonly used metaphorically for a closely related group of things—and specifically close enough to share the same keys. Runners up are group, collection and bundle, but those smell either too general or too overloaded.
2 Likes
Currently, to access a symbol in another module, that symbol needs to be declared
public. However, a symbol beingpublicallows it to be accessed from any module at all, both within a package and from outside of a package, which is sometimes undesirable.
crossmodule ?
1 Like
I'll have a larger reply tomorrow, but my initial response is, why not just change the design of spi so that it doesn't require ABI exposure in all cases? Can't we just take the mechanics that have already been written for this proposal and change how it interacts with the language to use a new form of SPI declaration rather than an access level?
Because ABI stability is a relied-upon feature of @_spi(). If you remove it, you wind up back at this situation:
1 Like
That’s why I said “in all cases”. Of course we can continue providing the public visible version but it would be enhanced with other visibilities.
One fundamental issue I have with this proposal is that it does not completely solve anything:
Afaics it is about helping teams organising a codebase that has become too bulky — but there is nothing that stops a library from growing even bigger.
So, what are you going to do when package hits the same limitations it is supposed to cure? Add a new superpackage level?
4 Likes
I tend to agree with this, it seems a very natural principle when working on code organization.
But if I understand the proposal correctly, some of the assumptions at its foundation are related to the fact that, given that only you or only your team work on the codebase, you can set up practices and conventions that would prevent cross-nuclei calls.
This might be true, but it could still be the case the 2 different teams work on 2 different nuclei, so a consequence of the assumptions would be that the 2 nuclei should be grouped in 2 separate packages, but that would then cause the issues that you mentioned, related to distribution (as a side note, this affects large distributed libraries, but not apps development, where we can simply create many local packages to organize code, so this proposal would still be helpful for us without major drawbacks).
So, it seems that the issue reduces to 2 options:
- either library distribution is made simpler and more convenient in case of 2 or more "internal packages" in a single library,
- or a better use for the
teamprivateaccess modifier would be to represent a group of modules within the same package.