⚠ Important packages are being released using unstable language features

While technically true, what you’re worried about is practically impossible. Apple would never replace such an attribute in a patch version, if only because they’d have to replace their own usage immediately. And given they’ve shipped public API with these attributes means the actual requirements for change, rather than the technical ones, are much higher than you think. We aren’t operating in a vacuum with only abstract rules.

But let’s say they do drop the attribute. The outcome is really no different: a new release to either adopt the replacement attribute or drop the usage altogether. Not really a big deal.

2 Likes

That's the point. People have this attitude that it's practically impossible for this stuff to be removed, so they use them even though they shouldn't.

Anyway - I leave it to the core team. They've been pinged. It's their problem, and their responsibility to ensure the package ecosystem is healthy and stable. For my part, I've done more than enough to warn everybody that unstable features are infecting the ecosystem. I'm tired of arguing with people.

3 Likes

FWIW, I agree with @Karl completely, and I find the use of underscored features in swift-numerics, swift-collections, et al a strong deterrent from using those packages in production code. It’s a pretty clear signal that ”while this package has a 1.0 version tag, it is not actually stable.”

1 Like

By that standard the Swift language itself isn’t stable, which is ridiculous. There’s been more churn due to bugs and public changes than these attributes, so I find attempts to point to them as examples of instability rather silly. This notion that libraries shouldn’t have to make releases to support new versions of Swift flies in the face of the entirety of the language’s history. Hyperbolic threads about these attributes do nothing to help anything.

8 Likes

I’m inclined to agree with @Karl on this one: underscored language features officially don’t exist, and are therefore undefined behavior. This is particularly problematic for alternative implementations of the Swift compiler (there are a fair number on various embedded systems, etc.), which must assume that only defined behavior needs to be implemented.

At the risk of kicking the can down the road, I think we may need different levels of support in Swift: things that should be assumed to break without recourse in major versions, but aren’t going to pop out of existence in minor versions.

Such things could sidestep the Swift Evolution process as a stopgap measure. For instance, _disfavouredOverload could get implemented under a different name in Swift 5.6, but compilers up to Swift 6 would accept it with a warning anyway. At that point it would just break.

Hopefully this would allow Swift to be a little more honest: some things need to be implemented messily, and that’s probably going to be true for quite a while. At any rate, it seems clear to me that the current state of affairs is unsustainable.

1 Like

I used @inline(__always) and @_exported import while creating ARHeadsetKit, ARHeadsetUtil, and MetalFFT. It would have been impossible to make those packages without the underscored keywords. Furthermore, unsafe flags in the Swift package manager were needed to pull off automatic differentiation in the Swift release build.

1 Like

If anything, I think the issue is that the language is moving too slowly on certain features that are extremely important in practice. I think these fall into three categories:

  • Memory/optimization control (@_effects, _modify, etc.)
  • Namespace control (@implementationOnly, @_exported)
  • Requirement tweaks (@_implements(ProtocolName, Requirement), @_unavailableFromAsync)

There needs to be an immediate focus on these issues, even if it means putting off wholly novel work like distributed actors. It’s getting out of hand.

As a start, the compiler should start requiring explicit arguments to use undefined behavior. This is what @_private(sourceFile: "FileName.swift”) does, and I think that’s sufficient disincentive to avoid misuse.

Second, we should start throwing the “unsafe” prefix around where necessary instead of pretending features with undefined behavior don’t exist.

Third, there needs to be less worry about renaming or changing things. Major versions should be expected to break things, and using underscores because a name might change or a future solution may be found is unacceptable. That is the only reason I can conceive of for some of these exclusions.

Finally, if something isn’t currently possible without language features that haven’t gone through evolution yet, do something else. Fork the Swift toolchain if you are sick of waiting.

1 Like

Finally, if something isn’t currently possible without language features that haven’t gone through evolution yet, do something else . Fork the Swift toolchain if you are sick of waiting.

Several of these features are needed on apps released to the iOS App Store, namely @inline(__always) for controlling performance. If they were forced into a separate (and not indefinitely maintained) toolchain like Swift for TensorFlow, that would seriously diminish how many applications could use them.

It also takes hours to build a Swift toolchain and a lot of expertise to get it right. That's not a feasible option for many people.

4 Likes

Then we need to acknowledge that they are part of Swift for the moment. Swift is supposed to avoid undefined behavior, for crying out loud!

The current state of affairs says a patch version of Swift can take every undocumented feature away at any moment. If that’s a problem, that must change.

Personally, I’ve found this extremely harmful to my ability to interpret Swift code written by others. I should be able to read the contents of Swift Algorithms without seeing code that officially shouldn't work.

1 Like

If you wish to take that position, fine, but you need to find another way to describe these features, as they certainly aren't undefined. Nor are they unsafe as Swift typically uses the term. At most they're unstable, but even that is stretching it given how heavily the language relies on them to get things done. Notably, this position: "I should be able to read the contents of Swift Algorithms without seeing code that officially shouldn't work." doesn't really make sense, any more than the use of other under documented Swift APIs "shouldn't work."

If we look at the history of these attributes and how they evolved, the @_functionBuilder to @resultBuilder transition is an obvious example. Technically, Apple could've dropped @_functionBuilder entirely once @resultBuilder was finalized, but that isn't what happened. Instead, a rather lengthy evolution process took place over multiple Swift releases and when it was finalized the old syntax was deprecated, not just removed. This helped not just the community that had adopted the attribute, but Apple's use of it in SwiftUI as well. Given how widely used some of these attributes are, and how important others are, it seems highly unlikely Apple would approach their evolution any differently, even if they could technically remove them tomorrow.

I do agree that the proliferation of such attributes is getting out of hand, especially for rather simple ones like @_disfavoredOverload. (At the least, such disfavor should likely be built into deprecation.) However, I'm not sure what the community can do to help here. Try to fully define particular features to prepare them for full evolution?

5 Likes

“Undefined” means they aren’t defined as part of the public interface, which means they can disappear in a patch version.

1 Like

That's not what "undefined" means in the context of programming languages. And they are part of the public interface, which is why they have to be underscored in the first place. It's not as if we're tricking the compiler into giving us access after all. Their "instability" is strictly name related.

3 Likes

If Swift isn’t SemVer-compliant, we might as well give up entirely.

Oh, come on - you know better than that, surely?

The standard library is not distributed in source form. It comes with one compiler, and if distributed, is already compiled.

If we're going to cross that line and start calling things "ridiculous", "silly", and "hyperbolic", that argument itself is ridiculous and betrays a severe lack of comprehension.

When have you ever seen the standard library being built by a compiler other than the one it ships with? The standard library's use of compiler hacks is a well-known special case -- and yes, those hacks are not at all stable!

If you're going to be so rude, try not to also be so wrong.

These attributes commonly show up in the language’s public module interfaces, making them a defacto part of Swift’s source stability story. Additionally, they’re used internally to provide critical functionality, like the performance guarantees of _modify. So the the impact of a change of either side largely the same.

2 Likes

Exactly. The issue here isn’t the standard library, or Apple frameworks that ship compiled with the OS. The issue is packages that ship as code and are compiled together with the code that uses them. That’s where the potential for breakage is.

I believe library consumers would want a tool to reject or accept libraries using unstable language features, regardless of their SemVer claims. That way, consumers can manage their own risk. A community-built tool could integrate with dependency-management and add metadata to package registries.

Aside from the value to consumers, outing libraries might discourage using unstable features. Conversely, permitting consumers to expressly opt-in might permit libraries to progress their versions to 1.0+ based on supported API's, with an unstable-feature proviso. And decisions about the unstable feature could be informed by data about its prevalence.

In either case, I think the core team has spoken, and noncompliance, however prevalent, is uncooperative and steals resources.

We need unstable features for experimenting, and we need stable libraries for scaling. I think consumers know which mode they're in, so they're in a good position to decide, and library authors can follow their consumers.

No, the issue exists in all code which uses these attributes, binary and source alike. A change to the public interface of the standard library would break compilation just like a change to the source of libraries. It's for exactly that reason that actual changes to these attributes are extremely unlikely without an upgrade path.

I would argue such a position is actually more unhealthy for Swift and the Swift ecosystem than the problem it purports to correct. Swift is not Rust. These are not strictly experimental features with well defined scopes. Many of these features, like _modify, are the only way to get Swift to work well in certain areas. Discouraging the use of them essentially freezes the ecosystem in very bad place for the language. Unless you think doing so would encourage the finalization of those features by Apple, which I don't, I can't see an upside to your proposal.

3 Likes

This is my hypothesis as well. We have Swift Source Compatibility that makes it trivial to spot compiler regressions. Note that NIO, Swift Collections, Alamofire, GRDB, RxSwift, and many more libraries that are widely used in the community are there. By registering in this list, the library authors made their part of the job.

As for underscored attributes used by GRDB, I can quote:

  • @_exported: Not strictly necessary, but it helps GRDB users use CocoaPods, SPM, or direct Xcode project integration in order 1. to access system SQLite, SQLCipher, or custom SQLite builds, or 2. change their installation method, without churn: just import GRDB and you're good to go.
  • @_disfavoredOverload: Required in order to work around compiler bugs such as SR-15150. I also see one usage of this directive commented with "Avoid a wrong resolution when BUILD_LIBRARY_FOR_DISTRIBUTION is set" in the GRDB source code, and I guess I could have filed a compiler bug for this one as well. Finally this directive is required in order to help the compiler resolve ambiguous apis.
  • @inline(__always): performance matters. Library authors profile their code, inspect assembly, and have to make decisions.

I also wish, of course, that the language would fix those holes as quickly as possible.

At the same time, I understand, of course, that it is a matter of allocation of resources.

I also understand that it takes time to fully understand what a feature is exactly about, and I'm not surprised those attributes are still underscored. We can all witness that the compiler is evolving independently of Swift evolution, and I'm pretty confident part of this silent work is setting up the stage for our requests.

Finally, it's difficult to ask the Core Team for commitment about the stability of those underscored attributes, because these underscores are their escape hatch. Practically speaking, the features they enable are there to stay, so I don't worry much. Everything @Jon_Shier has said is exactly what I foresee.

My final word is for library authors: please do register your code into Swift Source Compatibility!

5 Likes