Do these have corresponding issues filed on GitHub?
I'm not suggesting every single package should expose platform-specific traits, but I also I don't see how Embedded Swift is that different for packages that have to support Darwin, Linux, and Windows at the same time. Isn't that the same scenario essentially? If some packages already put effort into supporting multiple platforms abstracting these platform differences behind a portable API, same could be done for Embedded Swift.
Not sure if there is an "issue" to be raised for Codable. Metatypes and existentials simply are not supported, and Codable relies on them. I doubt there is an easy way out of this one...
It feels to me it is time for a Codable 2.0 anyway (probably macro-based, more compile-time generics magic, no reliance on existentials, embedded compatible) - but that is no small feat either...
this would be interesting to explore, but for me the difference between 56 MB and 3.7 MB is far more important than the difference between 3.7 MB and 500 KB.
56 MB is 20ish MB compressed, which is only viable in a very narrow set of circumstances. 3.7 MB is like 1.5 MB compressed, which is no worse a big PNG image and totally workable for many applications.
right now, i am principally concerned with finding a way around this nightly Swift compiler crash.
way ahead of you! swift-dom and swift-bson are already Foundation-free, neither library depends on FoundationEssentials, even. i have a deep catalog of Foundation-free libraries i have written over the years, so it feels like WebAssembly is a great opportunity for those investments to really pay off.
for JSON specifically, you might consider looking at swift-json, which has a non-Codable serialization system and is Foundation-free. its documentation is nowhere near as good as the docs for the BSON library, but the patterns are similar, if you have used the BSON library.
i tried cloning your repo, and building it as a whole with --swift-sdk 6.0.2-RELEASE-wasm32-unknown-wasi, and obtained a different error than the one you encountered:
/swift/wasm-test/galah$ swift build --swift-sdk 6.0.2-RELEASE-wasm32-unknown-wasi
Building for debugging...
warning: Could not read SDKSettings.json for SDK at: /home/ubuntu/.swiftpm/swift-sdks/swift-wasm-6.0.2-RELEASE-wasm32-unknown-wasi.artifactbundle/6.0.2-RELEASE-wasm32-unknown-wasi/wasm32-unknown-wasi/WASI.sdk
warning: Could not read SDKSettings.json for SDK at: /home/ubuntu/.swiftpm/swift-sdks/swift-wasm-6.0.2-RELEASE-wasm32-unknown-wasi.artifactbundle/6.0.2-RELEASE-wasm32-unknown-wasi/wasm32-unknown-wasi/WASI.sdk
<module-includes>:1:10: note: in file included from <module-includes>:1:
1 | #include "_CJavaScriptKit.h"
| `- note: in file included from <module-includes>:1:
2 |
<module-includes>:1:10: note: in file included from <module-includes>:1:
1 | #include "_CJavaScriptKit.h"
| `- note: in file included from <module-includes>:1:
2 |
<unknown>:0: warning: libc not found for 'wasm32-unknown-wasi'; C stdlib may be unavailable
warning: Could not read SDKSettings.json for SDK at: /home/ubuntu/.swiftpm/swift-sdks/swift-wasm-6.0.2-RELEASE-wasm32-unknown-wasi.artifactbundle/6.0.2-RELEASE-wasm32-unknown-wasi/wasm32-unknown-wasi/WASI.sdk
<unknown>:0: warning: libc not found for 'wasm32-unknown-wasi'; C stdlib may be unavailable
<unknown>:0: warning: libc not found for 'wasm32-unknown-wasi'; C stdlib may be unavailable
<unknown>:0: warning: libc not found for 'wasm32-unknown-wasi'; C stdlib may be unavailable
<unknown>:0: warning: libc not found for 'wasm32-unknown-wasi'; C stdlib may be unavailable
<unknown>:0: warning: libc not found for 'wasm32-unknown-wasi'; C stdlib may be unavailable
warning: Could not read SDKSettings.json for SDK at: /home/ubuntu/.swiftpm/swift-sdks/swift-wasm-6.0.2-RELEASE-wasm32-unknown-wasi.artifactbundle/6.0.2-RELEASE-wasm32-unknown-wasi/wasm32-unknown-wasi/WASI.sdk
<unknown>:0: warning: libc not found for 'wasm32-unknown-wasi'; C stdlib may be unavailable
warning: Could not read SDKSettings.json for SDK at: /home/ubuntu/.swiftpm/swift-sdks/swift-wasm-6.0.2-RELEASE-wasm32-unknown-wasi.artifactbundle/6.0.2-RELEASE-wasm32-unknown-wasi/wasm32-unknown-wasi/WASI.sdk
<unknown>:0: warning: libc not found for 'wasm32-unknown-wasi'; C stdlib may be unavailable
<unknown>:0: warning: libc not found for 'wasm32-unknown-wasi'; C stdlib may be unavailable
warning: Could not read SDKSettings.json for SDK at: /home/ubuntu/.swiftpm/swift-sdks/swift-wasm-6.0.2-RELEASE-wasm32-unknown-wasi.artifactbundle/6.0.2-RELEASE-wasm32-unknown-wasi/wasm32-unknown-wasi/WASI.sdk
warning: Could not read SDKSettings.json for SDK at: /home/ubuntu/.swiftpm/swift-sdks/swift-wasm-6.0.2-RELEASE-wasm32-unknown-wasi.artifactbundle/6.0.2-RELEASE-wasm32-unknown-wasi/wasm32-unknown-wasi/WASI.sdk
<unknown>:0: warning: libc not found for 'wasm32-unknown-wasi'; C stdlib may be unavailable
<unknown>:0: warning: libc not found for 'wasm32-unknown-wasi'; C stdlib may be unavailable
<unknown>:0: warning: libc not found for 'wasm32-unknown-wasi'; C stdlib may be unavailable
<unknown>:0: warning: libc not found for 'wasm32-unknown-wasi'; C stdlib may be unavailable
<unknown>:0: warning: libc not found for 'wasm32-unknown-wasi'; C stdlib may be unavailable
warning: Could not read SDKSettings.json for SDK at: /home/ubuntu/.swiftpm/swift-sdks/swift-wasm-6.0.2-RELEASE-wasm32-unknown-wasi.artifactbundle/6.0.2-RELEASE-wasm32-unknown-wasi/wasm32-unknown-wasi/WASI.sdk
error: link command failed with exit code 1 (use -v to see invocation)
wasm-ld: error: symbol exported via --export not found: main
clang: error: linker command failed with exit code 1 (use -v to see invocation)
warning: Could not read SDKSettings.json for SDK at: /home/ubuntu/.swiftpm/swift-sdks/swift-wasm-6.0.2-RELEASE-wasm32-unknown-wasi.artifactbundle/6.0.2-RELEASE-wasm32-unknown-wasi/wasm32-unknown-wasi/WASI.sdk
[420/422] Linking galah.wasm
this seems to be caused by the "-Xlinker", "--export=main" flags you added to GalahWeb in the package manifest. if i replace main with __main_argc_argv, like in Steven’s example, it builds successfully.
I just did some quick research and it seems like the change from main to __main_argc_argv is something that happened in Swift 6.0 (based off some comments in the JavaScriptKit codebase).
today i’m going to try rendering BSON to DOM, instead of to the JavaScript console.
i start coding, but i soon realize: i don’t want to keep using JavaScriptKit. instead, i want to use the WebAPIKit that @j-f1 recommended.
WebAPIKit is completely undocumented and has just one release. on the other hand i’m instantly drawn to how cleanly the code base is laid out, it looks like a high quality project well divided into many small modules.
i just have no idea how to use it.
WebAPIKit
i look at the package’s manifest, and it has an executable target called WebAPIKitDemo. so i look inside that. main.swift, to its credit, is a great example program:
the module i have to import is WebAPIBase, and i get it from a product also named WebAPIBase.
and i can’t build the project anymore because WebAPIKit has a module named DOM which collides with a module of the same name in swift-dom.
$ swift build
error: multiple packages ('swift-dom', 'webapikit') declare targets with a conflicting name: 'DOM’; target names need to be unique across the package graph
error: ExitCode(rawValue: 1)
i have to use a Module Alias, but that won’t really matter in WebAssembly since we are compiling everything from source anyway.
except i can’t actually import JavaScriptDOM for some reason.
/swift/wasm-test/Sources/BSONViewer/App.swift:1:8:
error: cannot refer to module as 'JavaScriptDOM' because it has been
aliased; use 'DOM' instead
1 | import JavaScriptDOM
| `- error: cannot refer to module as 'JavaScriptDOM' because it has been aliased; use 'DOM' instead
2 | import JavaScriptKit
carton doesn’t seem to understand the Module Alias at all.
warning: module alias for target 'DOM', declared in package 'wasm-test', does not match any recursive target dependency of product 'WebAPIBase' from package 'WebAPIKit'
error: multiple packages ('swift-dom', 'webapikit') declare targets with a conflicting name: 'DOM’; target names need to be unique across the package graph
warning: module alias for target 'JavaScriptDOM', declared in package 'wasm-test', does not match any recursive target dependency of product 'WebAPIBase' from package 'WebAPIKit'
error: multiple packages ('swift-dom', 'webapikit') declare targets with a conflicting name: 'DOM’; target names need to be unique across the package graph
error: ExitCode(rawValue: 1)
warning: module alias for target 'JavaScriptDOM', declared in package 'wasm-test', does not match any recursive target dependency of product 'WebAPIBase' from package 'WebAPIKit'
okay, so WebAPIKit is a no-go.
doing it the hard way…
SwiftPM makes me sad. if i comment out the WebAPIBase product dependency, but leave the swiftwasm/WebAPIKit dependency in place, i still get an error:
error: multiple packages ('swift-dom', 'webapikit') declare targets
with a conflicting name: 'DOM’; target names need to be unique
across the package graph
error: ExitCode(rawValue: 1)
which sort of hints that moduleAliases were never going to solve the problem in the first place, as moduleAliases appears at the target dependency level and not the package level.
but either way, the prognosis is the same: i will have to continue using JavaScriptKit to do things with WebAssembly in Firefox.
be lazy
there’s two broad strategies we could use to render BSON in the browser.
we could parse all the BSON recursively, and render the entire syntax tree to HTML, using <details> and <summary> to collapse hidden content.
we could parse the BSON lazily, only expanding nodes when the user clicks on them, generating DOM elements as needed.
is Swift WebAssembly fast enough to do Option 1? probably! i would hope? but it would probably still impose a ceiling on the size of the BSON this tool could handle, simply because Firefox would still have to parse and render all of that DOM.
have you ever tried to open a many megabytes JSON file in a JSON inspector? it’s slow as hell because JSON can only be parsed eagerly, even if you are only interested in drilling down into one particular node of the JSON.
and BSON was designed to support lazy parsing. this is, after all, the entire motivation for the format. so if we do Option 2, we could theoretically handle BSON that is many gigabytes in size, limited only by the amount of memory available to the WebAssembly.
i’m not going to go into too much detail with the actual code — i’ll upload the sample project to GitHub if anyone’s interested — but the basic approach i took was to have a protocol ExpandableAsDOM with a requirement expand(in:) that takes a JSObject.
types conforming to this protocol (BSON.Document and BSON.List) get an extension method called attach(toggle:container:) that adds a click handler to the toggle node.
now i want to preview this thing again with swift run carton dev --custom-index-page Web/index.html but one thing that i keep noticing happening is whenever carton rebuilds the project, sourcekit-lsp dies on me.
if i rebuild the project for the host and restart VSCode, sourcekit-lsp comes back, but then it dies the next time i run carton. so it’s almost as if they are fighting over IndexStoreDB or something.
when i click on the autonomousSystems node, which i know is a very large BSON list containing tens of thousands of elements, the app freezes up.
this is Encouraging, because i know i have built this WebAssembly in debug mode without release optimizations, and it confirms that my strategy of parsing BSON lazily was the right approach, because if i had tried to render the entire AST ahead of time, i wouldn’t be able to inspect this file at all.
how do i do WebAssembly in release mode?
now i’m kind of scratching my head, because it is not immediately obvious how to build the WebAssembly with release optimizations. i know there is swift run carton bundle, but that doesn’t seem to support previewing the application.
i try running swift run carton dev --help to see if there are any options i missed, but it doesn’t seem to recognize --help. at the top level, it does have --help, but nothing stands out to me.
$ swift run carton --help
Building for debugging...
[1/1] Write swift-version--75355AAB4E86B17C.txt
Build of product 'carton' complete! (0.38s)
Usage: swift run carton <subcommand> [options]
Available subcommands: bundle, dev, test, package, --version
i go back to pwsacademy/swiftwasm-examples because that helped me before, but the deployment examples in that repo seem to be pre-carton and i keep hearing that webpack Sucks and that carton is supposed to replace webpack.
(please, add some --help to the carton dev command)
previewing in release mode
armed with this new knowledge, i go to run swift run carton dev --custom-index-page Web/index.html --release and discover that my browser session has again been corrupted, so i open a new Private Browsing window, navigate to the app, and drop the same IP whitelist as before.
it’s slow — 160,000 DOM nodes is a lot for Firefox — but it Does Work!
cool! i’ve got a Minimum Viable Product now. we have definitively proved that The Idea Works and the Technology is Sound (?)
now i am thinking about how to package this as a Real App and deploy it to the internet for MongoDB nerds to play with. the carton README is pretty light on the actual details needed for how to do this.
so i take a look at the actual generated files to see if i can make sense of them.
$ ls -l Bundle/
total 4220
-rw-r--r-- 1 ubuntu ubuntu 4179379 Jan 23 23:09 BSONViewer.60b83c7e28769c34.wasm
drwxr-xr-x 3 ubuntu ubuntu 4096 Jan 23 23:08 JavaScriptKit_JavaScriptKit.resources
-rw-r--r-- 1 ubuntu ubuntu 56745 Jan 23 23:09 app.a71061e9b32ade2d.js
-rw-r--r-- 1 ubuntu ubuntu 337 Jan 23 23:09 index.html
-rw-r--r-- 1 ubuntu ubuntu 1619 Jan 23 23:09 index.js
-rw-r--r-- 1 ubuntu ubuntu 62313 Jan 23 23:09 intrinsics.js
-rw-r--r-- 1 ubuntu ubuntu 46 Jan 23 23:09 package.json
$ ls -l Bundle/JavaScriptKit_JavaScriptKit.resources/
total 4
drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 23 22:09 Runtime
$ ls -l Bundle/JavaScriptKit_JavaScriptKit.resources/Runtime/
total 56
-r--r--r-- 1 ubuntu ubuntu 27830 Jan 23 23:09 index.js
-r--r--r-- 1 ubuntu ubuntu 25118 Jan 23 23:09 index.mjs
as i understand it, carton expects you to use their index.html, which embeds a reference to app.a71061e9b32ade2d.js.
i don’t really like this, i expect to already have HTML, that may be generated dynamically, and i would want to be able to customize how the js file is served, which is made harder because it has a hashed name.
i would rather just embed that inline within the HTML, to eliminate a network round-trip. it’s not clear to me why all of the code above it is part of this file, and not part of the “Runtime” file instead. and despite the file extensions, none of the JavaScript seems to be meaningfully minified at all — index.mjs contains long comments and a lot of indentation.
conclusions
i wish i could have gotten WebAPIKit working, but i couldn’t figure out why moduleAliases was not resolving the module name collision problem. i would love to know what i am doing wrong there.
we really can achieve great performance with Swift WebAssembly, that unlocks architectures that simply weren’t viable with a traditional server and normal JavaScript. there is a lot of opportunity to do many things on the client side that used to be done server-side, and this will free up server resources to do server things and handle greater volumes.
i’m not sure how well carton’s commitment to file-watching and live-reloading will scale, as it kind of feels like it is cargo-culting workflows from the CSS/TypeScript world that just don’t make as much sense with a heavily optimized, compiled language like Swift. i suspect as Swift WebAssembly matures, we will fall back into the familiar build-launch-iterate workflow we use to develop other types of Swift projects — and this is okay!
Swift is not the only language that can compile to WebAssembly, but (compiler bugs notwithstanding) it is hard for me to understand why anyone would voluntarily choose C/C++ to target WebAssembly when Swift exists. a lot of the arguments against Swift today just don’t transfer over when targeting WebAssembly.
WebAssembly is one of the few areas where Swift’s compile-everything-from-source model doesn’t utterly kneecap the language outside of Apple’s ABI stable ecosystem. so i am cautiously optimistic that WebAssembly represents a huge growth opportunity for the Swift language and for this community. but to truly leverage this opening, we need to invest more in the Foundation-free library ecosystem, and seriously raise the bus factor for this technology, which is currently being held up heroically by an extremely small group of unpaid volunteers with seemingly no Support from Apple in over 2.5 years.
the Swift WebAssembly deployment story needs a lot of work. i would like to see the swiftwasm runtime distributed as a minified, self-contained JavaScript resource that can be hosted in a way that fits into the rest of a service architecture, and the “app.js” resource reduced to a tiny shim that is transparent to the developer and can be embedded, inlined, or even dynamically-generated any way they like. but i am hopeful that these rough edges can be polished with necessary attention.
Can't say for sure why they're fighting over things rather than the builds for different triples co-existing, but you can get SourceKit-LSP to see the same worldview as carton by telling it which Swift SDK to use.
Create a file in your project at .sourcekit-lsp/config.json with the following contents:
Reload VSCode, and SourceKit should now speak WASI!
Also wanted to note that development builds of the VSCode extension equivalently support configuring this via the swift.swiftSDK extension setting, which should also make its way into the next release.
This is intentional. Carton doesn't offer a lot of integration with existing web apps.
I wanted a more realistic approach, where you write the UI in JavaScript/TypeScript, not JavaScriptKit, and then just call into a Swift module for your business logic.
I tried building the Connect Four example using Carton and JavaScriptKit, but I wouldn't recommend going down that road. Code using JavaScriptKit is just as ugly and unsafe as actual JavaScript code, with even more complexities and less IDE support.
WASI imports and exports are probably the way to go, but they currently require complicated workarounds if you want to pass strings or structs. WASI Preview 2 should remove the need for those workarounds, but I have no idea when that will be supported in SwiftWasm.
I've added Embedded Swift support to my PR for IkigaJSON. It currently requires some manual hooks to parse the right data, and with minimal tweaking I could get my JSONObject and JSONArray types working on Embedded too. But I'm personally thinking of adding a macro-based Decoder here
Thanks for the discussions. FWIW we have been using (non-embedded) Swift Wasm in production since 2022 at app.flowkey.com (in the performance-critical part where you learn songs). We have a whole bunch of complex DOM behaviours, ML models and other DSP code, etc. last I checked the binary size was about 5mb compressed, but as you also touched on here Foundation can easily creep in and balloon that significantly (although nowhere near the 50+MB you saw in the thread). We also wrote WebWorkerKit to use distributed actors to achieve simple multithreading in our Swift code via web workers.
We are not using carton at all. We put Swift into an existing web app that builds via esbuild. We have a very simple plugin to achieve that which I have wanted to open source for years but have not gotten around to it. The main blocker is the complexity of WebWorkerKit which loads another copy of the binary in the worker: building this cleanly required some dirtier tricks for the esbuild plugin. If there’s interest I could look into this again though: I found carton great for simple use cases but I can’t imagine using it in production.
i spent some time today trying to reverse-engineer how carton generates the asset bundle from a Swift project, and i will summarize my findings below. (obviously, feel free to correct me where i am wrong.)
twice a year, the maintainers of carton run carton-release hash-archive, which launches esbuild and compiles the three TypeScript files bundle.ts, dev.ts, and intrinsics.ts. this process requires NodeJS to be installed on the maintainer’s development environment.
the intrinsics.ts is inlined into the other two files during the JavaScript compilation process.
the bundle.js file contains a sequence of characters 'REPLACE_THIS_WITH_THE_MAIN_WEBASSEMBLY_MODULE'.
carton-release produces compiled JavaScript in a temporary directory, which the release automation tool then reads, converts to Base64, and embeds as an enormous string literal in CartonHelpers/StaticArchive.swift. thus, carton-release is a sort of highly sophisticated macro that generates a format string that is embedded within carton itself and committed with the rest of the source code into the repo history.
the maintainers of carton seem to have tried very hard to do this in a more “sane” manner, but were hamstrung by the Swift compiler’s poor support for large static tables, so they had to resort to Base64.
at some point in the past, it seems this was also done for the JavaScript runtime, but this was abandoned because it forced everyone using a particular version of carton to also lock to a specific version of JavaScriptKit, and that was unacceptable.
when you run carton-frontend-slim bundle THE_WASM.wasm, it first loads the format string from inside itself, and substitutes REPLACE_THIS_WITH_THE_MAIN_WEBASSEMBLY_MODULE with the name of the wasm file, to generate the app.*.js. presumably it does this because SwiftPM does not have a better way of bundling static resources with executables on Linux.
it also reaches into the copy of JavaScriptKit that your app is using, and pulls out the index.mjs file which contains the JavaScript runtime matching the version of JavaScriptKit that you are using.
when you preview the app manually (python3 -m http.server 8000, not using carton), you can see it loads the app.*.js on the second round-trip, which then loads the index.mjs and the actual wasm on the third round-trip. none of the other generated files seem to be used as far as i can tell, so i am unsure as to their purpose in the generated bundle.
this produces a fairly suboptimal production bundle, because neither the app.*.js nor the index.mjs is properly minified, and carton cannot merge the two files into a single JavaScript resource because carton does not understand JavaScript.
i can sort of understand at an architectural level whycarton does it this way. the maintainers of carton do not want to depend on npm, and well, the reasons for why you would not want your tool to depend on npm should be self-evident . so my working hypothesis is that this extremely convoluted supply chain of pre-baked JavaScript files is motivated by not wanting users of carton to have to install NodeJS.
however, i would argue that this is Probably Not A Great Tradeoff. i think that to properly prepare a JavaScript bundle, you need access to a tool that understands JavaScript, and that means you need to depend on NodeJS.
As an ex-maintainer and original author of carton I can confirm, this was exactly the reasoning. The philosophy was to develop a zero-config solution for people who don't want to mess with JS dev tooling.
Accessing tools that understand JavaScript doesn't necessarily imply that users needs to depend on Node.js, there are enough tools and libraries outside of the Node.js ecosystem that can do it (esbuild is just one example). While there were no tools in the Swift ecosystem at the time, the plan was to wait and see until one appears or eventually to come up with our own tools for that.
And the other argument is that, if you have Node.js installed and you are willing to use it, why not adopt the whole Node.js stack, components of which would be better integrated with each other? At that point the only thing remaining for carton to do is toolchain and sysroot/SDK installation and management, but these days you can rely on Swift SDKs for that.
so i just tried retracing my steps on macOS (up to this point i had been using Ubuntu 24.04), and i found that the default Swift 6.0 toolchain on macOS does indeed fail to compile WebAssembly, with the same error you encountered.
it appears that you need to use one of the downloadable toolchains and switch to that by running export TOOLCHAINS=$(plutil -extract CFBundleIdentifier raw /Library/Developer/Toolchains/swift-6.0.3-RELEASE.xctoolchain/Info.plist).
i was then able to build a Hello World project by running swift build -c release --product Hello --swift-sdk 6.0.2-RELEASE-wasm32-unknown-wasi -Xswiftc -Xclang-linker -Xswiftc -mexec-model=reactor -Xlinker --export=__main_argc_argv.
What do you mean by "the default toolchain"? The Xcode toolchain? That one's only for targeting Darwin platforms. For cross-compiling to non-Darwin platforms you need swift.org toolchains, the other day I've updated information cards for these macOS toolchains to make the difference more obvious.
This is such an insightful breakdown of your journey into Swift WebAssembly! It's great to see the detailed steps you took, especially the struggle with installation and getting the SDK working. Your honest reflections and humor make it relatable to anyone starting out. Looking forward to seeing more of your progress!