How far can i get with Swift WebAssembly in 30 minutes?

[taylorswift]

@Max_Desiatov

For cross-compiling to non-Darwin platforms you need swift.org toolchains, the other day I've updated information cards for these macOS toolchains to make the difference more obvious.

yes, i realized that as i visited the page yesterday and that’s how i figured out that i needed a different toolchain (ha!)

to save everyone on macOS some time (as it gets asked a lot), i should warn folks the downloadable toolchains are built with compiler assertions enabled, even the “release” toolchains. therefore you will encounter many compiler crashes like the ones mentioned upthread. to avoid this, i would recommend using VSCode devcontainers on macOS and running a Linux image, or something equivalent.

[Max_Desiatov]

@taylorswift

i should warn folks the downloadable toolchains are built with compiler assertions enabled, even the “release” toolchains.

Is there any confirmation for this? From what I know, only developments snapshots have assertions enabled, while release tarballs don't.

[t089 (Tobias)]

This comes up regularly in the Swift slack, eg here Slack

macOS toolchains seem to always have debug assertions enabled

[Finagolfin]

@Max_Desiatov

From what I know, only developments snapshots have assertions enabled, while release tarballs don't.

I don't use the macOS toolchain, but several others who do use it have filed issues that they see assertion failures in the OSS 6.0.3 or earlier toolchains for macOS, ie not just the snapshots.

When I tried reproducing with 6.0.3 or other final releases on linux, I cannot, because assertions are disabled for such tagged releases on linux, but the same assertions trigger on the linux development snapshots. This leads me to believe they're right about the OSS 6.0.3 toolchain for macOS and earlier releases having assertions enabled, even though I've never used those macOS toolchains.

[taylorswift]

so it’s been a couple days since i last touched the project, and i came back tonight with the goal of getting something plausibly production-ready with the tools we have today.

but first, i discovered something very important regarding the binary size problems mentioned upthread!

Keep It Simple, Stupid

up until now, we thought the basic tradeoff of Swift WebAssembly was either: 1) ship a gigantic binary but be able to build complex projects that use Real Swift Libraries, or 2) ship a reasonably-sized binary but not be able to build anything non-trivial due to abundant compiler crashes.

@taylorswift

but the binaries are quite large. they take a very long time to download, which severely limits their practical applications.

these problems can be mitigated in theory by using experimental Swift compiler features, but this fails when you try to use Swift on WebAssembly to build anything bigger than a minimal test project.

it turns out this was never the Swift compiler, or the SDK’s fault at all! it was, in fact, carton that was injecting the massive +40 MB bloat into the compiled binaries.

here’s what i get when i run carton bundle at 1.1.3:

$ swift run carton bundle
Building for debugging...
[1/1] Write swift-version--75355AAB4E86B17C.txt
Build of product 'carton' complete! (0.78s)
- checking Swift compiler path: /home/ubuntu/.carton/sdk/wasm-6.0.2-RELEASE/usr/bin/swift
Inferring basic settings...
- swift executable: /home/ubuntu/.carton/sdk/wasm-6.0.2-RELEASE/usr/bin/swift
Swift version 6.0.2 (swift-6.0.2-RELEASE)
Target: x86_64-unknown-linux-gnu
Running "/home/ubuntu/.carton/sdk/wasm-6.0.2-RELEASE/usr/bin/swift" "package" "--triple" "wasm32-unknown-wasi" "--scratch-path" "/swift/bson-inspect/.build/carton" "--disable-sandbox" "plugin" "carton-bundle" "--output" "Bundle"
Building for debugging...
[1/1] Write swift-version-196F49A2765F111F.txt
Build of product 'carton-frontend-slim' complete! (0.63s)
Building "BSONViewer"
Building for production...
[0/2] Write swift-version-196F49A2765F111F.txt
warning: Could not read SDKSettings.json for SDK at: /home/ubuntu/.carton/sdk/wasm-6.0.2-RELEASE/usr/share/wasi-sysroot
warning: Could not read SDKSettings.json for SDK at: /home/ubuntu/.carton/sdk/wasm-6.0.2-RELEASE/usr/share/wasi-sysroot
[2/4] Compiling TraceableErrors NamedError.swift
[3/4] Compiling UnixTime Duration (ext).swift
warning: Could not read SDKSettings.json for SDK at: /home/ubuntu/.carton/sdk/wasm-6.0.2-RELEASE/usr/share/wasi-sysroot
[4/5] Compiling BSONABI BSON.AnyType.swift
warning: Could not read SDKSettings.json for SDK at: /home/ubuntu/.carton/sdk/wasm-6.0.2-RELEASE/usr/share/wasi-sysroot
warning: Could not read SDKSettings.json for SDK at: /home/ubuntu/.carton/sdk/wasm-6.0.2-RELEASE/usr/share/wasi-sysroot
[5/7] Compiling BSONEncoding Array (ext).swift
[6/7] Compiling BSONDecoding Array (ext).swift
warning: Could not read SDKSettings.json for SDK at: /home/ubuntu/.carton/sdk/wasm-6.0.2-RELEASE/usr/share/wasi-sysroot
[7/8] Compiling BSONArrays BSON.BinaryArray.swift
warning: Could not read SDKSettings.json for SDK at: /home/ubuntu/.carton/sdk/wasm-6.0.2-RELEASE/usr/share/wasi-sysroot
[8/9] Compiling BSON BSON.BinaryShapeError (ext).swift
warning: Could not read SDKSettings.json for SDK at: /home/ubuntu/.carton/sdk/wasm-6.0.2-RELEASE/usr/share/wasi-sysroot
warning: Could not read SDKSettings.json for SDK at: /home/ubuntu/.carton/sdk/wasm-6.0.2-RELEASE/usr/share/wasi-sysroot
warning: Could not read SDKSettings.json for SDK at: /home/ubuntu/.carton/sdk/wasm-6.0.2-RELEASE/usr/share/wasi-sysroot
[9/13] Compiling BSONReflection BSON.AnyValue (ext).swift
[10/13] Compiling DOM Character (ext).swift
warning: Could not read SDKSettings.json for SDK at: /home/ubuntu/.carton/sdk/wasm-6.0.2-RELEASE/usr/share/wasi-sysroot
[11/14] Compiling JavaScriptKit JSArray.swift
[12/14] Compiling HTML HTML (ext).swift
warning: Could not read SDKSettings.json for SDK at: /home/ubuntu/.carton/sdk/wasm-6.0.2-RELEASE/usr/share/wasi-sysroot
[13/15] Compiling BSONViewer App.swift
[13/15] Write Objects.LinkFileList
warning: Could not read SDKSettings.json for SDK at: /home/ubuntu/.carton/sdk/wasm-6.0.2-RELEASE/usr/share/wasi-sysroot
[14/15] Linking BSONViewer.wasm
Build of product 'BSONViewer' complete! (15.88s)
Right after building the main binary size is 59.06 MB

After stripping debug info the main binary size is 50.61 MB


Running wasm-opt -Os --enable-bulk-memory --enable-sign-ext /swift/bson-inspect/Bundle/main.wasm -o /swift/bson-inspect/Bundle/main.wasm
wasm-opt -Os --enable-bulk-memory --enable-sign-ext /swift/bson-inspect/Bundle/main.wasm -o /swift/bson-inspect/Bundle/main.wasm
`wasm-opt` process finished successfully
After stripping debug info the main binary size is 45.17 MB

Copying resources to /swift/bson-inspect/Bundle/JavaScriptKit_JavaScriptKit.resources
Bundle successfully generated at /swift/bson-inspect/Bundle

and here’s what i get when i just build the project directly with the SDK, without going through carton at all:

$ swift --version
Swift version 6.0.3 (swift-6.0.3-RELEASE)
Target: x86_64-unknown-linux-gnu

swift build -c release --swift-sdk 6.0.2-RELEASE-wasm32-unknown-wasi -Xswiftc -Xclang-linker -Xswiftc -mexec-model=reactor -Xlinker --export=__main_argc_argv

[33/33] Linking BSONViewer.wasm
Build complete! (22.78s)

$ ls -l .build/wasm32-unknown-wasi/release/BSONViewer.wasm 
-rwxr-xr-x 1 ubuntu ubuntu 11111971 Jan 30 23:41 .build/wasm32-unknown-wasi/release/BSONViewer.wasm

the completely unoptimized output of swift build is less than a quarter the size of the output of carton bundle with all size optimizations applied to it.

lesson learned: Keep It Simple, Stupid

do the trains run on time?

i tried @kabiroberai ’s suggestion to add this to .sourcekit-lsp/config.json:

{
    "swiftPM":
    {
        "swiftSDK": "wasm32-unknown-wasi"
    }
}

it seems to Work, but only when the project is being built in debug mode. building the project in release mode will not make syntax highlights appear, no matter how many times i reload the VSCode window. weird!

organizing the code

the next thing i did was start organizing my code into files, as it had previously been shoved into one single workspace while i was still in the “f***ing around” stage of the project.

luckily, this princess didn’t roll over any new peas during this step, although looking over the code did remind me of how much i wish WebAPIKit (or alternatively, SwiftPM module aliases) worked.

please, consider renaming those interior modules!

creating a bundle

the other day i wrote a tool called Archer which basically pulls out the JavaScript shim from the Carton Project, and just uses that along with WasmTransformer to serve as a dramatically stripped-down WebAssembly toolchain.

i’ll make a separate thread introducing Archer, but for those interested in following along, Archer has prebuilt binaries for macOS and Linux you can download from its README.

i used archer init to set up a frontend bundle:

$ archer init -o Bundle
esbuild --bundle Bundle/.archer/carton.js --outfile=Bundle/main.js --format=esm --minify

  Bundle/main.js  35.6kb

⚡ Done in 13ms

hint: copy a 'main.wasm', an appropriate 'index.html', and any other necessary resources to this directory, and run

    esbuild --servedir=Bundle

to preview your application in a browser

and then i used archer crush to add the WebAssembly to the bundle:

$ archer crush .build/release/BSONViewer.wasm -o Bundle/main.wasm

Size of original WebAssembly binary: 10.33 MB
Size after stripping debug symbols: 6.47 MB
wasm-opt -Os --enable-bulk-memory --enable-sign-ext Bundle/main.wasm -o Bundle/main.wasm
warning: active memory segments have overlap, which prevents some optimizations.
Size after optimization: 4.14 MB
output written to Bundle/main.wasm

of course, since i wrote the tool i can’t really meaningfully critique it. i thought archer Worked Great For Me. but it might be Totally Unusable for someone else. very keen on hearing any feedback others might have about archer!

generating index.html

Real Apps get Real Updates, so the HTML layer is going to have to be at least somewhat dynamic, because the URL of main.wasm, and probably main.js as well, will change with each update.

i don’t want to do cache busting, and i really don’t want to write literal HTML and send it through SwiftSoup.

all of this is going to rely on some kind of automated versioning, but fortunately SwiftPM (supposedly) has support for this since Swift 6.0.

getting the Git version

before i can use Git versioning, i need to turn this into a Git repository. that’s not too interesting, so i won’t live blog that. instead, i’ll just link the interesting pieces below and publish the thing to GitHub so everyone can take a look for themselves.

Package.swift

Package.swift

Sources/_GitVersion/swiftpm_git_version.c

Sources/_GitVersion/include/swiftpm_git_version.h

ugh, SwiftPM

the workflow i had imagined did not really work out too well in practice, for two reasons.

the first is that you can’t really run anything you build as WebAssembly on the host (at least not without fiddling with custom runtimes), which means if you have helper tools written in Swift you are going to be very sad because all of the SDK settings operate at the package scope. you have to build the helpers for the host and the app itself for WebAssembly, and neither SwiftPM nor sourcekit-lsp is well-equipped to handle that.

this problem is surmountable, but slows you down considerably, because coding without sourcekit-lsp kind of feels like driving down a highway at night with all the streetlamps turned off.

the second problem is that Context.gitInformation is pretty broken. i had gotten it working in two other projects previously, so i had felt pretty confident going in, but i just couldn’t get it to work here despite clearing every SwiftPM cache i could think of.

surely, there must be a rational explanation for why SwiftPM cannot pass the Git information, but there are already several known issues related to this, and i was so tired after fuzzing SwiftPM’s environment for so long trying to get it to work properly that i decided this would just have to wait for another day.

what i would have done differently

ultimately, i think i got Too Fancy and placed Too Much Faith in SwiftPM working properly.

the end goal would have been to run this in GitHub Actions as part of a workflow that pushes artifacts to S3 autonomously. GitHub Actions already knows about tag versions and could have passed us a version string as a command line option, and i could have used that to bypass Context.gitInformation entirely.

2020!

[dimi (Dimitri Bouniol)]

Here is my workaround for getting GitInformation via a simple package plugin, since the official means seem pretty non-functional:

Package.swift

        .plugin(
            name: "GitInformationPlugin",
            capability: .buildTool,
            dependencies: [
                .target(name: "GitInformation"),
            ]
        ),
        .executableTarget(
            name: "GitInformation",
            dependencies: [
                .product(name: "ArgumentParser", package: "swift-argument-parser"),
            ]
        ),

Plugins/GitInformationPlugin/GitInformationPlugin.swift

import PackagePlugin

@main
struct GitInformationPlugin: BuildToolPlugin {
    func createBuildCommands(context: PluginContext, target: any Target) async throws -> [Command] {
        let generatedSourceURL = context.pluginWorkDirectoryURL.appending(path: "GitInformation.swift", directoryHint: .notDirectory)
        return [
            .buildCommand(
                displayName: "Loading Git Information",
                executable: try context.tool(named: "GitInformation").url,
                arguments: [
                    "--root-path",
                    context.package.directoryURL.absoluteString,
                    "--generated-source-path",
                    generatedSourceURL.absoluteString
                ],
                outputFiles: [generatedSourceURL]
            )
        ]
    }
}

Sources/GitInformation/GitInformation.swift

import Foundation
import ArgumentParser

@main
struct GitInformation: ParsableCommand {
    @Option
    var rootPath: URL
    
    @Option
    var generatedSourcePath: URL
    
    func run() throws {
        let tag = try loadTag()
        let commit = try loadCommit()
        let isModified = try loadIsModified()
        
        let output = """
        enum GitInformation {
            static let currentTag: String? = \(tag.map { "\"\($0)\"" } ?? "nil")
            static let currentCommit: String = "\(commit ?? "")"
            static let hasUncommittedChanges: Bool = \(isModified)
        }
        
        """
        
        try output.write(to: generatedSourcePath, atomically: true, encoding: .utf8)
    }
    
    func loadTag() throws -> String? {
        let (returnStatus, output) = try git(path: rootPath, arguments: "describe", "--exact-match", "--tag")
        guard returnStatus == 0, let output else { return nil }
        
        return String(output.dropLast())
    }
    
    func loadCommit() throws -> String? {
        let (returnStatus, output) = try git(path: rootPath, arguments: "rev-parse", "HEAD")
        guard returnStatus == 0, let output else { return nil }
        
        return String(output.dropLast())
    }
    
    func loadIsModified() throws -> Bool {
        let (returnStatus, output) = try git(path: rootPath, arguments: "status", "--porcelain")
        return returnStatus == 0 && output != nil
    }
}

extension URL: @retroactive ExpressibleByArgument {
    public init?(argument: String) {
        if argument.prefix(7) == "file://" {
            if let url = URL(string: argument) {
                self = url
            } else {
                return nil
            }
        } else {
            self = URL(filePath: argument, directoryHint: .inferFromPath)
        }
    }
}

func git(path: URL, arguments: String...) throws -> (returnStatus: Int32, output: String?) {
    let process = Process()
    process.executableURL = URL(filePath: "/usr/bin/git", directoryHint: .notDirectory)
    process.arguments = ["-C", path.standardizedFileURL.path(percentEncoded: false)] + arguments
    
    let output = Pipe()
    process.standardOutput = output
    process.standardError = nil
    
    try process.run()
    let outputData = try? output.fileHandleForReading.readToEnd()
    
    process.terminate()
    process.waitUntilExit()
    
    return (
        returnStatus: process.terminationStatus,
        output: outputData.map { String(decoding: $0, as: UTF8.self) }
    )
}

[bbrk24]

@taylorswift

and despite the file extensions, none of the JavaScript seems to be meaningfully minified at all — index.mjs contains long comments and a lot of indentation.

The “m” in mjs doesn’t stand for “minified” (minified JS is usually, but not necessarily, .min.js). It stands for “module”. .mjs files can use import and export statements to share things between files. In regular .js, all files share the same global scope, so one file’s globals are visible — and overwritable — by any other file.