Carton: watcher and builder for SwiftWasm apps

Max_Desiatov · June 18, 2020, 8:33pm

I'm very delighted to announce that the first version of carton is ready to use. carton is a development tool for your SwiftWasm apps that watches your code, builds it when it changes, and serves the resulting WebAssembly binary via an HTTP server.

It also bundles a WASI polyfill, which is currently required to run any SwiftWasm code, and the JavaScriptKit runtime for convenience. The development version of the polyfill establishes a helper WebSocket connection to the server, so that it can reload development browser tabs when rebuilt binary is available. This brings the development experience closer to Xcode live previews, which you may have previously used when developing SwiftUI apps.

carton does not require any config files for these basic development scenarios, while some configuration may be supported in the future, for example for complex asset pipelines if needed. The only requirement is that your Package.swift contains at least a single executable product, which then will be compiled for WebAssembly and served when you start carton dev in the directory where Package.swift is located.

The main motivation for carton came after I had enough struggles with webpack.js, trying to make its config file work, looking for appropriate plugins. I'm convinced that the required use of webpack in SwiftWasm projects could limit the wider adoption of SwiftWasm itself. Hopefully, with carton you can avoid using webpack altogether.

So far carton dev works relatively well in my projects, and it doesn't require Node.js to be installed (as compared to webpack). You only need the official Swift 5.2 toolchain installed to build carton itself, and the SwiftWasm toolchain to build your projects with carton, see the installation guide for more details. carton is built with Vapor, SwiftNIO, swift-tools-support-core, and OpenCombine, and supports both macOS and Linux. (Many thanks to everyone supporting and maintaining those projects!)

Since a subset of Foundation and XCTest already work and are supplied in the latest snapshots of SwiftWasm SDK, the next top priority for carton is to allow running your XCTest suites directly in browsers and receiving test results back to the HTTP server, so that test results can be reported in CLI. This was blocked by XCTest not allowing customized test report formats, which is now partially resolved with a new argument available on XCTMain and a custom JSON test reporter. After a few other issues are resolved with the SwiftWasm toolchain, I'll get carton test fully working with XCTest.

There are a few more commands on the roadmap to be implemented, such as carton init for initializing basic SwiftWasm projects (potentially with configurable templates), carton bundle to produce an optimized production deployment bundle, SwiftPM resources support for bundled assets, carton sdk command for SDK installation and management and much more.

As cross-compiling to WebAssembly and running apps and tests remotely is not too dissimilar to Android development, or even development on macOS for Linux through Docker, carton could potentially become a generic tool for cross-platform Swift developers. I'm not developing any Android apps currently, but if there are interested Swift for Android developers, I'd be very happy to review and merge their contributions enabling that.

As they say at certain events "we can't wait to see what you build with this". Any feedback, criticism and contributions are much appreciated!

IOOI · June 18, 2020, 9:41pm

I wonder what would be possible if you'd combine carton and this: GitHub - carson-katri/SwiftWebUI: SwiftUI with support for WebAssembly into one product!

Max_Desiatov · June 18, 2020, 10:01pm

I'm not very familiar with SwiftWebUI internals to do that, only because I've been maintaining a basic implementation of SwiftUI API for WebAssembly in Tokamak, before the WebAssembly fork of SwiftWebUI became available.

Nevertheless, I don't think combining either of those with carton into one product makes sense. There are enough use cases for SwiftWasm without SwiftUI. One could use the plain DOM API with JavaScriptKit directly, or call into Canvas/WebGL with a hypothetical game engine library, which wouldn't match the SwiftUI API very well. Currently, carton supports all of these use cases, in addition to both Tokamak and SwiftWebUI, just because it's an independent developer tool.

It's the same reasoning why Xcode and SwiftUI aren't combined into one product, their purposes are different, and Swift development isn't limited only to SwiftUI.

ddunbar · June 21, 2020, 1:53pm

Would it make sense to decouple cartons watch and rebuild and serve from WASM? That feature is also super useful for any SwiftPM based server developer, right?

Max_Desiatov · June 21, 2020, 7:43pm

I wouldn't mind decoupling "watch and rebuild", that part is probably about a couple hundred LoC (the whole carton codebase is just ~900 lines currently). Where would it be decoupled to? Would it be a separate tool, or a part of SwiftPM?

The "serve and reload" part is probably too specific to carton and SwiftWasm, we use specifically Vapor (because it already has a WebSocket server included) with a few hardcoded routes. The front-end part with the WASI polyfill, JavaScriptKit runtime, and a WebSocket subscription is embedded too and is very specific to this use case. Maybe there's a good way to inject the WebSocket subscription script, but I guess it would take some effort to make it work reliably for an arbitrary rendered HTML page.

Max_Desiatov · June 26, 2020, 3:10pm

carton 0.2.0 is now available, which is able to install the SwiftWasm toolchain and SDK automatically for you if needed when you build your project with carton dev. You no longer need swiftenv installed thanks to this. This dramatically simplifies the whole installation process of SwiftWasm, where to use SwiftWasm and carton you previously had to:

Install swiftenv
Set up swiftenv by adding it to your PATH and adding it to your shell config
Install SwiftWasm with swiftenv.
Install carton.

With carton 0.2.0 this is no longer needed, you can now remove swiftenv if you don't use it for other purposes. Just install carton and start developing with carton dev in your project. The correct version of SwiftWasm will be installed automatically if/when needed.

Check it out on GitHub:

Max_Desiatov · July 8, 2020, 12:14pm

carton 0.3.0 adds a new carton sdk versions subcommand, new --release flag and a new --destination option to the carton dev command. Additionally, archive size is logged when a new SDK is downloaded, and backtrace logging is improved in browser consoles for crashing SwiftWasm apps. Many thanks to @RayZhao1998 and @ratranqu for their contributions!

Max_Desiatov · July 23, 2020, 11:40am

carton 0.4 is a big release and is available in Homebrew taps near you. Installing it on macOS 11.15 is as easy as brew install swiftwasm/tap/carton if you have Xcode 11.4 or later. Many thanks to Carson Katri, Ziyuan Zhao, Jaap Wijnen and Sergej Jaskiewicz for their contributions!

New features

Firstly, carton dev no longer requires a --destination flag with a manually crafted destination.json file to link with Foundation. If your project has import Foundation anywhere in its source code, a subset of Foundation provided with SwiftWasm is automatically linked. Please check the list of Foundation types currently unavailable in SwiftWasm for more details on Foundation compatibility (mostly filesystem, socket, multi-threading, and APIs depending on those are disabled).

The new carton init command initializes a new SwiftWasm project for you (similarly to swift package init) with two templates available at your choice. carton init --template tokamak creates a new Tokamak project, while carton init --template basic (equivalent to carton init) creates an empty SwiftWasm project with no dependencies.

The new carton test command runs your test suite in the wasmer environment. Unfortunately, this currently requires a presence of LinuxMain.swift file and explicit test manifests, --enable-test-discovery flag is not supported yet. Projects that can build their test suite on macOS can use swift test --generate-linuxmain command to generate this file.

The full changelog for this release is available on GitHub, and as usual any feedback is very much appreciated.

Max_Desiatov · August 20, 2020, 11:31am

carton 0.5.0 is now available on Homebrew with brew install swiftwasm/tap/carton or brew upgrade if you had it previously installed. This release updates both basic and tokamak templates in carton init for compatibility with the latest JavaScriptKit and Tokamak versions. Additionally, carton dev now cleans build logs from previous builds when its watcher is triggered. New --verbose flag was added, which restores the previous behavior with all build logs listed on the same screen.

carton dev and carton test now install 5.3 SwiftWasm snapshots by default, which in general are more stable than the previously used SwiftWasm development snapshots, and are compatible with Xcode 12 betas. You can now also add direct dependencies on a specific JavaScriptKit version instead of a revision with these 5.3 snapshots, as they contain a workaround for the unsafe flags issue reproducible with SwiftWasm development snapshots.

Allowing carton to select a default snapshot is now the recommended approach, so in general we suggest avoiding .swif-version files in projects that use carton.

The issue where carton dev hangs on exit after establishing at least one WebSocket connection with a browser is now fixed in our Vapor dependency. Kudos to @tanner0101 for diagnosing and fixing the issue!

Thanks to @carsonkatri and @RayZhao for their contributions to this release!