Pitch: support specifying Wasm features in package manifests

Necessary Disclaimer

Some may consider this pitch as premature, as the upstream Swift toolchain doesn't have full support for WebAssembly yet. The problem is that SwiftWasm team doesn't have enough capacity to submit all of the changes to the compiler by the time the next version of Swift is branched off.

At the same time, we already have a few packages that we're actively using with the SwiftWasm toolchain and allow other people to depend on. Additionally, the SwiftWasm build of SwiftPM has very little (and soon hopefully none at all) differences from upstream SwiftPM. We'd like changes that enable Swift support for WebAssembly to land in SwiftPM first. This will allow us to support more WebAssembly features in the near future without fragmenting the whole Swift ecosystem.

wasm as a Supported Platform in Package Manifests

As a quick introduction, I need to mention that WebAssembly is a collection of multiple proposals that are at different implementation stages in WebAssembly hosts. The bare minimum is called WebAssembly MVP and is available in all major browsers. Other features such as atomics, SIMD, reference types and many more are not as widely available (with Safari lagging behind in most of these).

The SwiftWasm team would be happy to start experimenting with atomics, SIMD, and reference types when that is possible. Atomics is a big one, as it unblocks multi-threading support with big parts of core libraries that are currently unavailable in SwiftWasm. We could then make Dispatch and major parts of Foundation available in browsers.

I don't think we'll be able to do that transparently to our users. I'm convinced that use of atomics should be opt-in, at least per package. This would allow package authors to preserve compatibility with older browsers. It's obviously applicable to the rest of WebAssembly features, so we need a way to specify this in package manifests.

My pitch here is to introduce a new SupportedPlatform.WasmFeatures struct conforming to OptionSet in the PackageDescription module.

extension SupportedPlatform {
    /// The supported WebAssembly features.
    public struct WasmFeatures: Encodable, OptionSet {
        public init(rawValue: Int64) {
            self.rawValue = rawValue
        }

        /// The underlying features representation.
        public let rawValue: Int64

        /// Minimum available feature set.
        static let mvp = WasmFeatures([])

        /// WebAssembly System Interface is available.
        static let wasi = WasmFeatures(rawValue: 1 << 0)

        /// Shared linear memory and atomic memory access are available.
        static let atomics = WasmFeatures(rawValue: 1 << 1)

        /// Fixed-width SIMD is available.
        static let simd = WasmFeatures(rawValue: 1 << 2)
    }
}

Not all Wasm features are specified here as some of them are too low-level. I've only picked a few that could be useful in the near future, but the list can be easily expanded (as long as we don't have more than 64 features and run out of bits in the rawValue property).

This way our users would be able to specify features they require in Package.swift:

// swift-tools-version:5.4
// the Swift toolchain version that supports this is assumed to be 5.4,
// but is not guaranteed
import PackageDescription

let package = Package(
    name: "SwiftWasmApp",
    platforms: [.wasm([.atomics, .simd])]
    // other package settings omitted here for brevity
)

Alternatives considered

Keeping the status quo

We could keep the situation as is, but I'm convinced this would be detrimental to the user experience. We could also wait until WebAssembly support is fully available in the upstream Swift toolchain, but that could put barriers to development and adoption of the SwiftWasm ecosystem. If we discuss this pitch as early as possible and it is accepted, we hope for the implementation to land in the current merge window. If it is rejected, we'll get the necessary feedback and start working on alternatives early enough for this to be resolved soon.

Specifying versions of Wasm hosts and browsers

We could allow specifying browsers and their versions directly as supported platforms:

// swift-tools-version:5.4
// the Swift toolchain version that supports this is assumed to be 5.4,
// but is not guaranteed
import PackageDescription

let package = Package(
    name: "SwiftWasmApp",
    platforms: [.chrome(68), .safari(15), .firefox(79)]
    // other package settings omitted here for brevity
)

In this case I guess some part of the toolchain (or just carton itself in the short term) would have to maintain a database of what features are available in what browser versions. But in the end that would require more infrastructure changes to work. Additionally, specifying Wasm features and browser versions is not mutually exclusive in principle. .wasm(.simd) could be useful for non-browser WebAssembly hosts, while .firefox(79) would imply a JavaScript environment and DOM availability, which is also useful to encode in platform requirements.

If the community considers specifying browser versions in the list of support platforms useful, I would prefer this to be discussed as a separate pitch.

Source-level checks

The are different levels of feature checks to discuss. We could also add support for @available attributes and if #available runtime checks at the source code level. As we see, Apple platforms support all of these, and what would be the reason to not support them for other platforms? Package-level checks in SwiftPM are easy to implement though (I have a PR for it in the SwiftWasm fork), while the source-level checks should be discussed separately. It still would be interesting to gauge the interest in the community for any of these.

This change is additive, so I don't think it could harm other platforms that Swift supports in some way.

Thanks for reading, and I look forward to your feedback!

19 Likes

My understanding is that "core libraries" have to be available on all official toolchains. It's worth considering if a wasm host that does not support Dispatch or major parts of Foundation even qualifies as a supported platform.

The pitch is about broadening support for WebAssembly in SwiftPM only, not the whole upstream toolchain, as I mentioned in the disclaimer. Upstream SwiftPM already has support for .wasi platform, which is especially handy with conditional target dependencies introduced in Swift 5.3. It allowed us to write libraries supporting the SwiftWasm toolchain and compatible with the upstream toolchain at the same time.

There are many other platforms where supporting all "core libraries" wouldn't be feasible (AVR and any other bare metal platform). I hope we'd rather consider other options as "micro" runtimes and subsets of stdlib stripped out for platforms with constraints, which would greatly broaden portability of Swift in general. Or maybe something akin to tiered platform support in Rust.

As I wrote in the other thread, we at SwiftWasm would very much like to strip as much as possible (and core libraries too) for certain apps because of binary size constraints. But what the upstream toolchain considers as "a supported platform" shouldn't have much if any impact on this pitch, as rudimentary support for WASI is already there in upstream 5.3 SwiftPM.

I don’t think @Karl here is disparaging the validity of WebAssembly as a platform; rather I think it goes to the question of this particular proposal, whether it makes sense to talk about a SupportedPlatform with options that toggle the requirement for threads or a libc equivalent.

If the intention is to start exploring stripped-down runtimes, then that’s a much, much bigger conversation than here. Suffice it to say that Swift doesn’t have that support today.

1 Like

I think it does. I'm open to considering the other option with explicit browser versions, but listing Wasm features makes as much sense. I wish Wasm was properly versioned, with MVP being .v1, atomics supported added in .v2, and SIMD added in .v3 with browsers and other Wasm hosts implementing it in that exact order. Then this would neatly match what SupportedPlatform does for Apple platforms. We would then mark the Dispatch module unavailable for packages that don't require .wasm(.v2) and SIMD module unavailable for packages that don't require .wasm(.v3).

This isn't the case unfortunately. All Wasm hosts (including browsers) pick different features they'd like to implement in arbitrary order, and makes perfect sense in this open ecosystem. Listing all Wasm hosts in SwiftPM explicitly is unfeasible. I think that plainly listing (some) Wasm features is more scalable.

Browsers warrant a different approach as there are only 3 major browser engines, and 4 major browsers if we count Edge separately from Chrome. They provide access to JavaScript for SwiftWasm apps, and that's useful for packages to specify.

So I'd personally prefer SupportedPlatform.WasmFeatures option set for all Wasm hosts, while browser platforms can be added separately if the community is open to it. But these two approaches would serve different purposes. Some packages might need SIMD w/o access to JavaScript/DOM, some could be designed purely for browsers.

Stripped-down runtimes are not a subject of this proposal, I was only addressing Karl's point about core libraries. Happy to discuss that separately if needed.

The alternative I have in mind (and perhaps @Karl, but he can speak for himself) isn’t explicit browser versions, but rather whether it makes sense instead only to support WebAssembly that comes with atomics and WASI. In other words, I’d propose we consider that feature set as the “minimum viable” platform for the purposes of Swift, even as the MVP for the purposes of browsers is less. (I don’t mean abandoning any support for other configurations while bringing up the implementation of Swift on WebAssembly, but as a question of the final design.)

The reasons for this consideration are severalfold. Principally, though, is that the MVP from the perspective of browser support is orthogonal to what makes for a usable or complete experience for Swift users. Just like Windows platform support doesn’t include (and, indeed, because of what’s needed to support core libraries, can’t include) Windows 7/8 support, we are not obliged to support all “versions” of WebAssembly just because we support some version of it.

We are fortunate that, with most users now on some flavor of Chrome, Firefox, Chromium Edge, or Safari, browser support is constantly improving at a greater pace than Swift’s minimum requirements for supported platforms are expanding. It is not clear to me that the MVP shipped in browsers today will be particularly relevant even in the broader (outside-of-Swift) web ecosystem by the time that WebAssembly support is upstreamed into Swift. So the question is, will supporting what may be a historic artifact by then be worth the engineering effort to design this and other APIs now? I’m not so sure.

My point in bringing up the issue of pared-down runtimes in Swift generally is that, if Swift’s minimum requirements for supported platforms were already re-engineered to be more modular, then it would change the calculus here. But it is not, and I think the example of how we handled Windows requirements is a good precedent to consider here for WebAssembly. I would be entirely supportive of growing Swift’s WebAssembly support to include MVP-only implementations in tandem with growing Swift’s support for embedded platforms, etc.; I think that requires a systematic and larger effort though, which support for some configuration of WebAssembly currently congenial to Swift’s requirements does not.

3 Likes

Or, put another way perhaps, in bringing up a supported platform, we need to think about our (Swift’s) MVP.

Supporting Swift on WebAssembly MVP goes beyond what’s necessarily the MVP for Swift on WebAssembly (which would make more demands of the platform). We didn’t do it for Windows, and it’s not done for Linux either (I think we still don’t support musl instead of Glibc). It seems by tackling WebAssembly’s MVP upfront and upstream that we are inverting priorities here.

1 Like

I'm all for it, but the lagging support in Safari makes this very hard to consider. It doesn't look like atomics and threading support in WebAssembly is on the Safari/WebKit development roadmap at all, and improvements to WebKit end up in Safari with a very significant delay. Therefore, even if atomics and threads somehow magically start working in WebKit today, there's a very slim chance it will be supported in Safari 15 on iOS 15 and macOS 11.1 (or whatever the versions next year will be). And then we'd have to wait a few more years for all Safari users to upgrade their OS, devices, and browsers for this to get signifcant adoption.

This would mean that Swift for WebAssembly would be unavailable to a very significant amount of users (basically all iOS users and some macOS users that use Safari) until at least 2022-2023. That's under assumption that Safari/WebKit implements these features quickly if at all. And past performance shows that it usually doesn't.

If assuming atomics will be available in all browsers (eventually) is realistic, I'm not aware of any plans for browsers to support WASI directly. WASI is and will be in the foreseeable future supported in browsers through a polyfill, which is quite heavy. If anything, WASI is a stopgap measure for the SwiftWasm project to get things going, and we'll drop our dependency on it as soon possible.

The plain JavaScript polyfill is at least 300kb optimized, while WASI-libc that we link statically into the binary is much heavier than that. This overhead makes the use of SwiftWasm questionable in many consumer apps. Because of our WASI dependency, we can't be compared favourably if at all to other competitors in this space such as AssemblyScript, which has a mere 2kb runtime overhead.

I also don't think there's much code in Swift runtime and stdlib that requires WASI. I hope that with a custom allocator and a replacement for ICU we could drop it and strip down the binary size quite significantly. Then the requirement for WASI to be available on Wasm hosts will become irrelevant.

I don't want this discussion to drift away from the original pitch of a trivial additive change to SwiftPM to a more philosophical discussion of pared-down runtimes in the upstream Swift toolchain. The proposed change would be consistent with the existing wasi platform in SwiftPM. It would unblock the SwiftWasm team on a lot of fronts and wouldn't require any changes in the upstream Swift compiler whatsoever. We can continue discussing minimal runtimes for Swift, either as a part of WebAssembly support or for other platforms. But it's a big separate topic that's related to the upstream toolchain and core libraries, not SwiftPM per se.

To elaborate on this a bit, some of the focus here was on WASI and threads, but there are many more interesting features that we'd like to support at some point. Interface types are coming to WebAssembly sooner or later, and this will completely change the way we interact with code imported from hosts. Similarly, 64-bit memory will be there too. But when they're released, we don't want Swift for WebAssembly to drop support for Wasm hosts that don't have these features. At the same time we want to allow people to use them if they want to.

This is not much different from deploying to iOS 11 when developing an iOS app that also runs on iOS 14. Some APIs are specified as unavailable on older iOS, and some Swift packages require the latest iOS to work, some don't. Right now this can be specified in Package.swift.

We want the same thing for WebAssembly, not much more. It would work with our forked SwiftWasm toolchain, and it would work in the future when we merge the compiler and stdlib changes upstream. The only difference is that WebAssembly doesn't have monotonically increasing versions, but an arbitrary mix of clearly separated and specified features. OptionSet type for Wasm features as a platform version specifier seems to be a good way to express that. But the upstream SwiftPM needs to support it for this to be interoperable across the whole Swift ecosystem.

I want to emphasize that the pitch touches only SwiftPM, source-level checks and changes to @available attributes that require compiler support are out of scope of this pitch.

Yes, this is what I meant as well.

If wasm is unable to support the core libraries, and thus unable to support many popular Swift libraries, it is unlikely to become an officially supported platform any time soon. It just won't work with most Swift code.

I understand that you've set this as a goal, but my understanding is that it is a "nice to have". It is not a requirement to not use WASI - indeed, my understanding is that you're using it in your toolchain right now (as you say, a stopgap measure - but it works for an MVP).

So then it becomes another feature entirely - about how to run on stripped-down platforms without an OS, or how to require/use optional hardware features like the various SIMD extensions and atomics. There has been a lot of interest in that topic, but it's nothing we need to solve urgently before 5.4 AFAIK.

With iOS versions, SwiftPM allows the use of a custom version string. This permits users to specify versions before they officially exist. Can this be sufficient for WebAssembly to unblock your work? For instance, could we consider "multi-value reference-types" to be a “version string” for these purposes (with all the “webbiness” of a user-agent string :face_vomiting:)? This would have the advantage that we would minimize the API surface area.

I think I would prefer a design where the desired features and derived browser requirements are a result of the invocation instead of the manifest.

swift build --triple wasm32-unknown-wasi \
   --experimental-wasm-features wasi,atomics,simd

SwiftPM would turn that into compilation conditions that apply to the whole graph. Each individual library can then do any of these:

#if EXPERIMENTAL_WASM_WASI
#elseif EXPERIMENTAL_WASM_ATOMICS
#elseif EXPERIMENTAL_WASM_SIMD
#endif

That way you can vend a library that takes advantage of features if they are available, but doesn’t necessarily require them.

It also makes a simpler path to deprecation if the features become so standard that enabling them all the time makes sense.

1 Like

The 5.3 release for Windows doesn't support SwiftPM, which means it doesn't work with most Swift code. Does that make Windows not an officially supported platform?

I'm a bit surprised that you mention that support in core libraries is a requirement for being an officially supported platform. I don't think I saw it mentioned by the Core team anywhere, did I miss anything? Is it mentioned anywhere on swift.org or toolchain docs?

The Swift Core Libraries project provides higher-level functionality than the Swift standard library. These libraries provide powerful tools that developers can depend upon across all the platforms that Swift supports.

https://swift.org/core-libraries/

As for Windows, AFAIK it is not incapable of supporting SwiftPM.

Obviously there is some leeway - even the standard library isn’t absolutely identical on all systems, but I don’t think we’ve encountered a situation of someone wanting an official port (which I assume you do) to a platform incapable of supporting major portions of the core libraries. I guess it’s ultimately up to the core team to decide.

Just to make my view of it clear: the platform supported by SwiftPM is “wasi”, not bare WASM. What you are proposing are WASM capabilities - which IMO is analogous to asking for specific hardware capabilities. For example, the ability to say that a module is only compatible with WASM hosts with SIMD extensions isn’t really different to somebody saying a module requires an x86 processor with AVX512 extensions. I believe @scanon has also expressed interest in something like that.

I would support a general feature to check for/use hardware extensions at runtime, and to allow packages to restrict their supported platforms based on those features.

Runtime checks are not ideal for browser apps, I'm primarily interested in compile-time checks here. I'd rather have two separate Wasm binaries with and without a feature enabled. Then I can send either of those over the wire based on a value in the User-Agent header, and either of those would be smaller than a single binary with a runtime check.

OK, so I've looked in to this a bit more, and this is what the situation in Rust seems to be, according to recent GitHub issues:

WASM features are just added ad-hoc via compiler flags in Rust. Different combinations of WASM features may not be binary compatible, so this also means that you need to build your own standard library for each combination:

Also as of today there is no dedicated target for wasm with atomics. The usage of -Ctarget-feature=+atomics was intended to help ship this feature ASAP on nightly Rust, but wasn't necessarily intended to be the final form of the feature. This means that if you want to use wasm and atomics you need to use Cargo's -Zbuild-std feature to recompiled the standard library.

(And yes, users do trip up on this).

The feature has been this way for the last 2 years. Their best idea seems to be similar to our idea of a "Swift MVP" - to create some kind of name for the bundle of WASM features we need:

AFAIK no progress has been made on shipping this in the standard library. The best idea we've had (which is the same as two years ago) is to have a second target (naming TBD) which is wasm-with-atomics.

Github issue

Moreover, I think we need to make sure that WASM atomics even has everything we need to port the core libraries like libDispatch:

Overall threads, wasm, and Rust I feel are not in a great spot. I'm unfortunately not certain about how best to move things forward. One thing we could do is to simply stabilize everything as-is and call it a day. As can be seen with memory initialization, imports, and TLS, lots of pieces are missing and are quite manual. Additionally std::thread has no hope of ever working with this model!

In addition to the drawbacks previously mentioned, there's no way for TLS destructors to get implemented with any of this runtime support. The standard library ignores destructors registered on wasm and simply never runs them. Even if a runtime has a method of running TLS destructors, they don't have a way of hooking into the standard library to run the destructors.

Given all of that, I think this pitch is premature.

1 Like

I do like the “target-feature” flag though, and I think something like that as an addition to SwiftPM manifests might be generally useful. For sufficient generality, it may have to be stringily typed as I commented above.

1 Like