Pitch: Standard Library Preview Package

Hi Swift community,

Here's a first draft of a proposal we've been working on that should help us work in our new, post-ABI stability, post-Xcode package manager integration world.

PR for the proposal can be found here.

Standard Library Preview Package

Authors: Ben Cohen, Max Moiseev

Introduction

We propose the addition of a new package, SwiftPreview, which will form the initial landing spot for certain additions to the Swift Standard Library.

Adding this package serves the goal of allowing for rapid adoption of new standard library features, enabling sooner real-world feedback, and allowing for an initial period of time where that feedback can lead to source- and ABI-breaking changes if needed.

As a secondary benefit, it will reduce technical challenges for new community members implementing new features in the standard library.

In the first iteration, this package will take the following:

  • free functions and methods, subscripts, and computed properties via extensions, that do not require access to the internal implementation of standard library types and that could reasonably be emitted into the client
  • new types (for example, a sorted dictionary, or useful property wrapper implementations)
  • new protocols, with conformance of existing library types to those protocols as appropriate

The package will not include features that need to be matched with language changes, nor functions that cannot practically be implemented without access to existing type internals or other non-public features such as LLVM builtins.

For the purposes of this document, we will refer to the proposed standard library preview package as "the package" and the standard library that ships with the toolchain as "the library".

Motivation

Facilitating Rapid Adoption and Feedback

It is common for a feature proposed and accepted through Swift Evolution and added to the Standard Library to wait for months before it gets any real life usage. Take SE-0199 (the introduction of toggle) for example. It was accepted almost 6 month before Swift 5 was released.

Even though all additions go through a thorough review, there is no substitute for feedback from real-world usage in production. Sometimes we discover that the API is not quite as good as it might have been.

While the toolchains downloadable from swift.org are useful for experimentation, they cannot be used to ship applications, so are rarely used for much more than "kicking the tires" on a feature. The beta period for Xcode provides slightly more real usage, but is still relatively short, with feedback often coming too late to be applied, needing, as it should, a further review on Swift evolution as well as time to integrate into the converging release. And the nature of standard library additions are such that you do not always have an immediate need early in the beta to try out a feature such as partial sort or a min heap.

Once a feature ships as part of a Swift release, any future changes resulting from feedback from real usage must clear the very high bar of source and ABI stability. Even if the change is merited and a source break is justified, the absolute need for ABI stability can rule out certain changes entirely on technical grounds. Furthermore, for performance reasons, standard library types that rely on specialization need to expose some of their internal implementation as part of their ABI, closing off future optimizations or performance fixes that are only discovered through subsequent usage and feedback.

Technical Challenges for Contributors

The requirements for contributing to the standard library can be prohibitive. Not everybody has the time and resources to build a whole stack including LLVM, Clang, and the Swift compiler itself just to change a part of the Standard Library. Additionally, Xcode and XCTest cannot easily be used to maintain and test standard library code.

Integrating changes into the standard library requires knowledge of non-public features and idioms relating to ABI-stability. In particular, implementing a performant, specializable, ABI-stable collection type while preserving partial future flexibility is significantly harder than writing one that is only source-stable. Harder still is knowing what internal implementation details can be changed after such a partially-transparent type has been published as ABI.

It would be of great benefit to the community to allow proposals and contributions to the standard library without these requirements, leaving integration of the final ABI-stable version to a later date and possibly other contributors with more experience maintaining ABI-stable code.

Proposed solution

We propose the introduction of a "Standard Library Preview" SwiftPM package.

It will be a standalone project hosted on Github under the Apple organization, and will be part of the overall Swift project.

Proposals for additions to the standard library will have the option of first landing in the package for a period before migrating to the library. A PR against the preview package will be sufficient to fulfill the implementation requirement for an evolution proposal.

Additions to the package will continue to use the Swift Evolution process. All additions to the package should be made on the assumption that they will in time migrate to the ABI-stable standard library module that ships as part of the Swift toolchain. Proposals first landing in the package should be given the same level of scrutiny currently applied to standard library additions today.

However, requirements around source stability of the package will not be the same as those of the Swift standard library. Follow-up proposals and proposal amendments will be able to make source-breaking changes. Unlike the very high bar for source-breaking changes, and the absolute rules around ABI stability, the bar for changes to API currently only in the package will be that of any other evolution proposal of a new API.

Starting life in the package will not be mandatory. Proposals can opt instead to go straight into the standard library, especially if they do not meet the criteria for suitable package additions (see detailed design section). Functionality that are already available elsewhere, and have been proven in real-life use in that way, and is being "sunk" down into the standard library, would also be good candidates to skip the preview stage.

Since the package will not be ABI-stable, it will not ship as a binary on any platform, or be a dependency of any ABI-stable package. This allows for changes to the internal implementation of any type, and the change/removal of any function, as part of implementation changes, follow-on proposal amendments, or subsequent proposals.

Detailed design

The following additive evolution proposals could be made using the preview package:

  • New algorithms implemented as extensions on library protocols
  • Utility types (e.g. wrapper types returned from functions like the Lazy types underlying .lazy)
  • New protocols, including conformances by standard library types
  • New collection types
  • Property wrappers, such as a late-initialized wrapper

The following are examples of changes that would not go into the package:

  • Types introduced as part of and inseparable from language features (like Optional or Error)
  • Implementations that rely on builtins for a reasonable implementation (like atomics or SIMD types)
  • Implementations that require access to other types internals to be performant
  • Changes that can't be done via a package, like adding customization points to protocols

Some of these cases will require a judgement call. For example, making an extension method a customization point may bring a minor performance optimization — and so not prevent initial deployment in the package — or it may be a major part of the implementation, making the difference between an O(1) and O(n) implementation. Whether a proposal should go into the preview package should be part of the evolution pitch and review discussions.

Migration

An important aspect of the design is the experience of users of the package when features migrate from the package to the library. This is handled differently for functions and types.

Types

The preview package will have a different module name (SwiftPreview) to that of the standard library (Swift). As such, every type it declares will be distinct from the type once it migrates into the standard library, and can co-exist with it. Because the preview package is "above" the Swift module in the stack, users of the package will get the package version of the type by default in source files that have imported the preview package. They will be able to specify the library version instead by prepending Swift. to the type name. This has the benefit that addition of the type into the library is source-compatible with code already using the package version.

It does have a downside for code size. Package users do not benefit from the code-size wins of not including the new type in their app and instead using a version in the OS (though, given that most types in the standard library are generic and need to be specialized, this is not as big a problem as it might be). It would also mean package users miss out on optimizations that may be possible with the library implementation. Once inside the library, fast paths could benefit from internals of other types. For example, a ring buffer might be implemented in terms of the same storage as an Array, and conversion from one type to another could just be a question of copying a reference to the other's storage.

In these cases where the standard library's version of the type would be better, the user can easily switch to it by adding a typealias TheType = Swift.TheType to their code, or by prefixing the module on individual declarations if needed.

Functions

Unlike types, methods cannot be disambiguated. That is, you cannot write something like myCollection.SwiftPreview.partialSort. Swift does not have this feature yet (and while desirable, it should not be a dependency of this proposal). So there is no way similar to the typealias approach above to prefer the standard library's implementation of a protocol extension over the packages.

Since Swift 5.1, the standard library has had an internal feature that allows it to force-emit the body of a function into the client code, as a way of back-deploying critical bug fixes. This @_alwaysEmitIntoClient attribute can be used from within the standard library to deploy functions only to prior platforms, at the cost of binary size of the client (when they use them). Again, this cost is mitigated in the case of protocol extensions by the fact that the specialized implementation is already inlined.

This allows use of #if compiler directives to simultaneously obsolete implementations in the package for the latest version of Swift when introducing new functions into the standard library. As long as the new library definitions are marked as @_alwaysEmitIntoClient the source compatibility of this should not be noticeable.

Type/Function Combinations

A common pattern used in the standard library is to return a type from a function, with the type driving much of the functionality initiated by the function call. For example, myArray.lazy.map returns a LazySequence which maps elements on the fly using the supplied closure.

Since types cannot be emitted into client code, these combination function/type features will follow the pattern for types. This will mean that the package version will be used by default, and type context will be needed to force a call to the standard library if desired.

Retirement from the Package

In order to keep the package size and maintainability manageable, implementations will be removed from the package in a version update 1 year after migration to the standard library. This is a necessary balance between maintainability and convenience. Since the package is open-source, it should be possible to copy source for a specific feature if a user badly needs to keep it while also wanting to upgrade to a newer major version and not upgrade to the latest compiler.

Evolution

Introduction of the package does not change much in the process of Swift Evolution. Changes to the Standard Library API surface should go through a well developed pitch - discussion - implementation - proposal - decision life-cycle.

The main difference is that the final result will not land straight into the standard library. Instead, at a later point, a new kind of review will have to be run — one to promote parts of the package to the library. This should be done via an amendment to the proposal, and should be kept lightweight. A timeline for this secondary review should be set at proposal acceptance. The duration of time spent in the package should reflect the complexity of the proposal, and thus the amount of "bake time" it might need.

No proposal should be accepted into the package on a provisional basis: it should be assumed that every proposal will be migrated as-is unless feedback while in the package stage reveals that it was a clear misadventure. The review manager of the migration amendment will be responsible for ensuring that the review discussion focuses on new feedback from real-world use, and not purely relitigation of previously settled decisions made during the original review.

Testing

The standard library currently uses a combination of StdlibUnittest tests and raw lit tests. This works well, but is unfamiliar to most developers.

The preferred style of test for Swift packages is XCTest, and the package should adopt this approach. This has the benefit of familiarity, and integrates well with Xcode. The work to convert tests from this format to lit can be done at the point of migration into the library.

Certain features of StdlibUnittest – in particular crashLater and StdlibCollectionUnittest's checkCollection, as well as the helper types like the minimal collections – should be ported over to XCTest if possible. This would make for an excellent starter bug, or could be done as part of introducing the first full-featured collection to the package that had a significant need for this kind of testing, but are not a requirement for accepting this proposal.

GYB

There will be no use of gyb in the package. Gyb is a powerful and useful tool, but it can cause code that is difficult to read and write, and does not interact well with Xcode, so runs counter one of the goals of this package: to simplify contribution.

The need for gyb has been gradually reducing over time as language features like conditional conformance and improved optimization eliminated the need for it. Generally speaking, most uses of gyb indicate missing language or library features, and so implementations of proposals should probably start by proposing those features instead. For example, it is unlikely we would accept further proposals to splat out multiple operators for different sizes of tuple like we do with ==. Instead, protocol conformance for non-nominal types and variadic generics should solve this problem more generally.

Versioning

The fundamental goals of the preview package are somewhat at odds with the usual goals of semantic versioning. Source-breaking changes, including retirements after migrating into the standard library, or source-breaking changes resulting from evolution proposals prior to migration, will be common. Unlike many packages, the preview package will not converge over time, reducing the frequency of its version bumps.

As such, the preview package will remain at major version 0 permanently. Minor version bumps will be able to include source-breaking changes. Patch updates should not break source and will just be used for bug fixes.

The use of a 0 major version at all times will act as a signifier that adopters need to be comfortable with the requirement to continuously update
to the latest version of the package. It should not be taken as an indication that the package shouldn't be used for real-world code: the code should be considered production-worthy. But it is a preview, and not source stable. Where practical, deprecated shims could be used to preserve source compatability between versions, but this may not always be practical.

The decision when to tag minor versions, and potentially to coalesce multiple changes into a single version bump, will be made by the release manager for the standard library as chosen by the Swift project lead.

Source compatibility

None on Swift itself. The intention of the package is to facilitate long-term source stability by detecting issues with APIs prior to their integration into the library.

Effect on ABI stability

None. The package will not be declared ABI stable. ABI stability will only happen when proposals are migrated to the standard library. This means that the preview package may not be used within binary frameworks.

Because we do not have cross-module optimization, package implementations should make use of @inlinable annotations. However, these annotations should be in the context of source stability only and should be fully reevaluated by the library integrator when stabilizing the ABI.

Alternatives considered

Single versus Multiple Packages

An alternative to a single monolithic preview package would be multiple packages. For example, each accepted evolution proposal could have its own package.

This would trade convenience for flexibility. In particular, multiple packages would avoid the source-breaking nature of proposal revisions (and the lesser problem of any additions being technically source breaking). But it would mean that adopters of the different previews would need to look up and add individual proposals rather then get them all in one go, and would rarely "discover" new features also in the package they're already importing.

Simultaneous Preview and Library Addition

If the goal was to facilitate rapid release only, but not a longer feedback period, then the additional review step would not be necessary. Instead, new features would be integrated directly into the package and library simultaneously.

However, experience has shown that proposals are not always perfect. Recently, significant if manageable issues were discovered with the API of both SE-180 and SE-202, and in a near-miss SE-220 had to be reverted late during 5.0 convergence due to typechecker performance impact. These issues would likely have been caught by a more prolonged exposure in a package.

Is a Second Review Necessary?

Whether or not a proposal accepted and integrated into the package actually needs a second review is also worth considering. An alternative would be to set a timer, and then to automatically migrate the proposal into the library when the timer expires, without need for a second review. Changes could still be initiated to the preview package, but a new proposal would be required to interrupt the otherwise automatic migration timetable.

84 Likes

Having read the title and first paragraph, then skimmed to the bottom skipping all the details, I am strongly in favor of this!

7 Likes

I think this will be a super welcome change for the evolution of the standard library. Excited to see this rolled out! A couple questions:

Should this read "...removed from the package in a minor version update..."? Later on the Versioning section says there will be no major version bumps.

This sentence was cut off, but I'm curious what the conclusion was going to be.

Oops yes. An earlier draft proposed constantly bumping the major version, but @johannesweiss convinced me the "staying at zero" convention would be a better plan.

Oops, thanks! This was meant to say the duration should reflect the complexity of the proposal, and thus the amount of "bake time" it might need.

3 Likes

How often would SwiftPreview be released?

A good point for discussion, but I would say it should be released continuously, with minor version bumps whenever proposals or amendments are accepted, rather than on a scheduled cadence.

24 Likes

@Ben_Cohen Is there currently / will there currently be a way to get sections of a module? Maybe this is a better question just about SPM functionality, but I know in some other package managers there are ways to only get subsets of all available packages.

For example, in Cocoapods you can have dependent packages using subspecs.

Could we potentially do something similar where new features are added in a way so that they can be pulled in individually, but are all referenced from a master package?
(I found this SPM pull request)

I think have the granularity to say which experimental features you'd like to try but also having a master spec to grab all new experimental features might be useful.

1 Like

I have to admit I'm not enough of a package expert to know if we could avoid these problems this way (with existing or future features). Maybe @Aciid or @johannesweiss have a view.

That said, my preference is not to hold off on instituting this new process based on future features that SPM might acquire.

Yay, +1, this is awesome!

Strongly +1.

Yes, you can vend multiple products in the package but it'll have to be in a different module. For example, there could be two modules in the package: SwiftPreview, SwiftPreviewExperimental

Given the goal and purpose of this package, I would suggest not releasing any semver tag and ask adopters to use branch-based dependencies instead. This will restrict the use of this package to the top-level application and help avoid dependency-hell situations where different packages end up depending on different versions of the preview package. You can even have a master and preview branch for distinguishing "development" and "released" code.

9 Likes

+1, agree with the previous point by @Aciid that using it based on branch names would be much better. It also would communicate a bit better. Given that we have a resolved file and there are commit-based requirements too, it's still possible to lock it down when needed.

Sounds great, I love it.

Something worth considering (maybe out of scope), is the asymmetry of adding and removing functionality.

Does Swift favor additive evolution, and does this lead to more complexity over time?

I think this is a great change either way, it should lead to more iteration before adoption, which hopefully means more refined APIs and simplicity.

It’d be good to try to find a subtractive equivalent (I have no solutions). A regular culling (simplification, deprecation) of APIs and syntax is healthy, maybe when major versions are bumped.

Edit: Could things be moved out of the standard library into this package as a form of deprecation?

Ooh, version-number-strategy bikeshedding, this is new!

Since Swift is an evolving language, we can expect that new language features will be added in the future. And, similarly, we can expect that subsequent proposals will make use of those future language features.

Therefore, any particular iteration of the Preview Package will have some minimum version of the Swift Language which it requires. This could be worked around through copious #if swift directives, but that seems unnecessarily tedious.

Perhaps we should have the major version number of the Preview Package reflect the version of Swift for which it is a preview.

4 Likes

That’ll be taken care of with swift-tools-version in the manifest file.

The evolution process supports deprecation as much as addition. We have frequently (perhaps too frequently for some) removed functionality through evolution proposals.

But: ABI is forever. Once something is in the ABI, it can never be removed. The most you can do is deprecate/obsolete. As such, there's not the same motivation to "preview" this, because no matter what, the symbols will remain, so you can always "un-obsolete" something later if you end up with regret.

2 Likes

This is probably too heavy-handed a restriction. While many packages may choose to avoid using the preview for pragmatic reasons, it is perfectly reasonable for a package to say "we use the very latest preview package and that means you have to too if you want to use us".

4 Likes

This looks great. There are some bits that I've not had time to fully push through the proposal process which would be good fits for this secondary avenue of experimentation, exposure and discussion.

To be clear, as currently pitched this would not be for experimental changes. The expectation is proposals remain something that is agreed as an addition to the standard library – it just gives us some breathing room for feedback just in case there are issues uncovered in real-world use.

The best way to experiment will remain to stand up a new package, outside of the swift evolution process, and get people to try it out. This is a great way to try out changes before proposing them to evolution. Depending on the extent of adoption of that package and its maturity, it might make sense to still stage them into the preview package when then putting them through evolution, or they could go straight into the regular library.

5 Likes