Can you clarify how exactly these subdivisions could be infinite? An API can already be restricted within a module with internal, and within a single file with private or fileprivate. What other subdivisions would one ever be interested in other than these?
Not every module, but only those that use package, if I understand your question correctly. Moving code around would require deleting package keywords only when such code is moved outside of package, say to a single-file script, which in practice is quite rare. Off the top of my head I can't say I've done it at least once in my career, and I've been writing in Swift as my primary language almost immediately after it became available.
Can you elaborate how exactly this would be source breaking?
Agree that this seems less-than-ideal. I’m not sure I see a strong motivation for making this an error — it doesn’t seem actively harmful, just redundant / unnecessary. Maybe it could just be a warning instead?
I understand that, I was making the point that this proposal allows for a division into packages one level deep, but doesn't solve the general case of subdividing modules into very precise access restrictions.
I also understand this proposal does not suggest reverting a keyword. I was trying to state that by introducing package immediately, we may be introducing another fileprivate scenario which we cannot seemingly revert.
I went through the proposal text, and sorrily didn’t see package’s advantage over SPI except for a shorter syntax. Most “bonus”es from the pitch can simply apply 1:1 to the SPI world.
The biggest problem of this pitch, IMO, is that it assumes an arbitrary package context for Swift codes, which is absolutely wrong because there’re plenty of Swift codes in the form of standalone script files, CMake(-compatible) projects, Xcode projects, standalone XCTest suites, etc. All of them are not necessarily (for some, almost never) “package”s. I really don’t like the idea of “introducing a keyword in the Swift language, but for SwiftPM packages only”.
What somehow makes things worse is, as @8675309 has pointed out, splitting giant packages (projects) is a best practice in software engineering, but this pitch makes it way harder to do that while SPI just works smoothly.
Below are some other nits of the proposal, which just didn’t change the overall -1 I would give:
After this pitch, the “exported interface” is divided into two layers: public and package, which could have different module dependencies. That means in order to maintain the package-level interface, a target may rely on (and even accidentally expose!) some modules that are totally unused in public APIs, and vice versa.
Also, try to consider “qualified imports” — what if we want to explicitly import only public APIs inside the package for some purpose (eg. providing an example/template target for users)?
@elsh, can you comment on the intended behavior here? It would be good to understand sooner rather than later if the proposal needs to be updated to accommodate the full range of package identities support by SwiftPM. Alternatively, if the intent is that SwiftPM will only enable package for SwiftPM packages that conform to the schema described in SE-0292, the proposal should make that clear so that we can get adequate feedback on the idea.
From a compiler perspective, I have a couple comments:
Treating what's clearly readable text as simply an arbitrary sequence of bytes seems sketchy from a Unicode correctness standpoint. Some filesystems do this for long-term stability reasons, but it seems harder to justify here. I suppose the assumption is that the build system will pass down a stable canonicalization of the string?
I don't know what SwiftPM's "internal identities" are, but (1) passing them to the compiler makes them not purely "internal" anymore, so they would have to be rigidly defined and stably encoded, and (2) future directions like automatically namespacing package-private modules by package name would definitely prefer that the string be relatively stable and compact, rather than including a lot of local paths and other URL fluff.
Names including invalid c99 identifier characters are currently encoded using the rules specified at spm_mangledToC99ExtendedIdentifier. Product and target names are converted to a valid c99 identifier with those rules, as seen here; "Προϊόν" is converted to "Προι_ο_ν".
We will allow the full range of package identities supported by SwiftPM. Will update the proposal accordingly.
The proposal is not arguing that SPI is an unnecessary concept that shouldn't be added to the language; it's just arguing that SPI is not the right tool in every situation. The SPI feature is designed around promoting a tight coupling between a particular interface and its exact expected clients. Sometimes you want that, and in those cases, you should use SPI. However, sometimes you don't want that, and all you want to say is "this interface is just for us, at least for now". Swift's core access control design is based around recognizing existing boundaries that often correspond to a useful sense of "us": the people implementing this specific declaration, or working in this file, or working on this module. In that light, allowing users to recognize a boundary that's broader than a single module but not as broad as the entire program makes sense, because small teams often work on several closely-related modules and should be able to share code without having to make it part of the public API. Different organizations can draw that boundary where they want, but it makes sense for SwiftPM to default to drawing it at the package boundary, and package seems like a good general name for the feature.
It would certainly be more flexible to allow these boundaries to be explicitly named and separately controlled. If you want that, you can use the SPI feature for it. But personally, I think that would not be a good idea. It really needs to be notable for a file to contain an SPI import: seeing one should make programmers and code reviewers immediately question whether it's really needed. That idea is badly undermined if SPI imports are common; if every file contains two or three boilerplate SPI imports just to get access to run-of-the-mill cross-module interfaces, it becomes really easy to miss that one of them is a "true" cross-package SPI usage that should get special attention. In my mind, the only way to avoid that and preserve the value of programmer discipline and code review is to design access control the way Swift does, around ever-larger natural boundaries that programmers intuitively understand. And ultimately, access control within an organization only works because of programmer discipline and code review, because otherwise programmers will just make things public to shut the compiler up.
I don't understand what this conversion is doing. Προι_ο_ν is not a "C99 identifier" in the usual sense of an identifier that is required to be accepted by conforming C99 implementations. Is there a misunderstanding here about what "C99 identifier" means?
It's hard to keep public-facing modules in a package from never having internal or package declarations. While it's useful to have package-private modules, having a package access modifier also gives a public facing module flexibility, e.g. it could have its own helper function in the module that is package and need to include a package symbol from another module in its signature; if a public function were to include it, it would not be allowed.
Okay, but "c99 extended identifier" is not precisely defined. The C99 standard throws up its hands and just says it's implementation-defined what other characters are allowed in identifiers. It certainly doesn't say that Greek characters are allowed as long as they don't have accents. That's just what your function does, it's not a specification.
That mangling seems reasonable to use, at least by SwiftPM.
It is a public method that would have been package if this proposal had existed way back when it was written, because it is an implementation detail. The method is for mangling an arbitrary string into something that will be accepted as an identifier by the entire toolchain. “c99” might be an over‐simplified name. I think the result preserves c99 explicitly allowed characters, including c99 implementation defined characters (i.e. excluding c99 explicitly disallowed characters), but the actual transformation of the illegal characters is an arbitrary implementation detail.
I only meant that the name’s structure can be an implementation detail (albeit pan‐toolchain) and not an API promise unless we are deliberately trying to support compiling something halfway with SwiftPM and halfway with Bazel. Whatever tool is arranging the build graph can decide on its own identities and mangling. (Some sort of mangling is obviously wise if there is worry about Unicode changing while passing between precesses, but if the compiler is doing its own, then it is liable to trip the higher level tool.)
If we do want all client tools to interoperate, then we need to nail down the mangling as an API condition. It would narrow SwiftPM’s ability to add new features without direct compiler acknowledgment. It would also constrain sister tools like Bazel to match SwiftPM’s model more closely.
The proposal does talk briefly about "sub-modules" in the Alternatives Considered section. I'd be interested in your thoughts about what you think sub-modules would be and how they would be a better feature. In general, I feel that vague proposals sometimes seem attractive precisely because they are vague; they can shift to meet any goals we have at the moment without ever needing to compromise. So it is useful to explore what sub-modules would actually be like in order to understand their implications for the language and their suitability for solving this language problem. Are sub-modules required to be modular, i.e. independent units with acyclic dependencies? Do they create a namespace, and if so, is that namespace exposed as part of the interface of the containing module? How does one specify that something is accessible only within the sub-module vs. the containing module?
A simpler, somewhat similar, solution to some aspects of this problem would be what I call "sub-targets", which would be similar to CocoaPod's subspecs. That is, rather than forcing package authors to split into modules or not split at all, allow parts of a target to be offered separately at a source level. For instance, I could break Alamofire multipart form encoding functionality into a separate sub-target that isn't included by default, simply by putting the functionality in a separate source file. Since I don't want to offer that functionality as a separate, public module, nor some internal module (no matter the form) I wouldn't have to make any source changes, as the functionality would never care it may not always be compiled.
Of course, this doesn't solve the problem of being able to label and separately import the parts of my target, but if the author of the package doesn't want that functionality, letting them split targets, and letting users consume parts of the target, would allow at least some of the advantages of granular consumption without having to formalize your intertarget boundaries.
Of course there are also some things SPM would need to do to keep integrity, like requiring only the whole target to create ABI stable artifacts (this needs to be updated anyway, as SPM really should be able to create a platform's preferred stable distribution form), but it seems a workable solution for smaller packages.
@_spi would also not be easy to optimize. By design, clients of an SPI can be anywhere, making it effectively part of the public ABI of a module. To avoid exporting an SPI, the build system would have to know about that specific SPI group and promise the compiler that it was only used in the current built image. Recognizing that all of the modules in a package are being linked into the same image and can be optimized together is comparatively easy for a build system and so is a much more feasible future direction.
It seems like the issues with SPI could avoided by formalizing it and allowing a visibility modifier as part of the SPI declaration. Couldn't we simply mark an SPI as package visible to gain these same advantages? That way we gain both a formalized SPI feature (which seems necessary) and don't have to change the access modifiers at all. I could even see this update to SPI enable things like compiler-enforced testing only APIs, or any other visibility we want, without having to modify other parts of the language at all.
In fact, it seems like we could combine access levels for SPI and allow things like package and testing only APIs at the same time, a feature just not possible with access levels without major changes.