Issue using Swift macros in SwiftPM package when targeting Wasm

Tl;dr

I'm failing to compile a SwiftPM package for WASM due to the package's use of macros. Is this expected behaviour? And is there a known workaround?

Cheers!

Context

I'm writing an interpreter for a custom programming language (galah) and I'm attempting to build the interpreter for WASM so that I can make an online playground for people to try out the language (not much to see yet, the language is still pretty simple and very constrained).

In the interpreter's code I've used macros to express a pattern that I couldn't otherwise (the details aren't important).

The issue

I have attempted to build the interpreter for WASM with both of the following commands,

/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2024-04-04-a.xctoolchain/usr/bin/swift build --experimental-swift-sdk 5.10-SNAPSHOT-2024-04-09-a-wasm --target GalahInterpreter

/Users/stackotter/Library/Developer/Toolchains/swift-wasm-5.9.2-RELEASE.xctoolchain/usr/bin/swift build --triple wasm32-unknown-wasi --target GalahInterpreter

In both cases I get the follow errors,

/Users/stackotter/Desktop/Projects/LangDev/galah/.build/checkouts/swift-syntax/Sources/SwiftCompilerPlugin/CompilerPlugin.swift:103:19: error: cannot find 'dup' in scope
    let inputFD = dup(fileno(stdin))
                  ^~~
/Users/stackotter/Desktop/Projects/LangDev/galah/.build/checkouts/swift-syntax/Sources/SwiftCompilerPlugin/CompilerPlugin.swift:117:20: error: cannot find 'dup' in scope
    let outputFD = dup(fileno(stdout))
                   ^~~
/Users/stackotter/Desktop/Projects/LangDev/galah/.build/checkouts/swift-syntax/Sources/SwiftCompilerPlugin/CompilerPlugin.swift:124:11: error: cannot find 'dup2' in scope
    guard dup2(fileno(stderr), fileno(stdout)) >= 0 else {
          ^~~~

I have run the commands with -v and found that SwiftPM is building SwiftCompilerPlugin for WASM even though it's only getting used by macro (which afaik gets compiled for the host platform, so compiling SwiftCompilerPlugin for WASM is completely unnecessary work, and incorrect if incorporated into the macro).

Package setup

// swift-tools-version: 5.9

import CompilerPluginSupport
import PackageDescription

let package = Package(
    name: "galah",
    platforms: [.macOS(.v10_15)],
    products: [
        .library(
            name: "GalahInterpreter",
            targets: ["GalahInterpreter"]
        )
    ],
    dependencies: [
        .package(url: "https://github.com/apple/swift-argument-parser", from: "1.3.0"),
        .package(
            url: "https://github.com/apple/swift-syntax.git",
            from: "509.0.0"
        ),
        .package(
            url: "https://github.com/stackotter/swift-macro-toolkit",
            from: "0.3.1"
        ),
    ],
    targets: [
        .executableTarget(
            name: "galah",
            dependencies: [
                "GalahInterpreter",
                .product(name: "ArgumentParser", package: "swift-argument-parser"),
            ]
        ),
        .target(
            name: "GalahInterpreter",
            dependencies: [
                "UtilityMacros"
            ]
        ),

        .macro(
            name: "UtilityMacrosPlugin",
            dependencies: [
                .product(name: "SwiftSyntax", package: "swift-syntax"),
                .product(name: "SwiftSyntaxMacros", package: "swift-syntax"),
                .product(name: "SwiftCompilerPlugin", package: "swift-syntax"),
                .product(name: "MacroToolkit", package: "swift-macro-toolkit"),
            ]
        ),
        .target(name: "UtilityMacros", dependencies: ["UtilityMacrosPlugin"]),

        .testTarget(
            name: "GalahInterpreterTests",
            dependencies: ["GalahInterpreter"]
        ),
    ]
)

Troubleshooting

I tried splitting the macros into a separate (local) package (to remove the direct swift-syntax dependency from galah), but got the exact same errors again.

2 Likes

SwiftPM does not support Macro with cross-compilation context in the stable toolchains yet.
But the support has been landed quite recently, so the next nightly toolchain will unlock the limitation hopefully. Revert "Revert "Support macros when cross-compiling (#7118)" (#7352)" by MaxDesiatov · Pull Request #7353 · apple/swift-package-manager · GitHub

2 Likes

Amazing, thanks! Had a feeling it might be something WIP that I hadn't heard about.

I'll try building main branch SwiftPM for myself and try it out before the next toolchain.

1 Like

Thanks for the help! After resolving a few other issues that I ran into with JavaScriptKit and stuff, I finally managed to get an online playground working :tada:


I faced a few different issues while trying to get this working, if any of these aren't already known I'd be happy to split them out into separate topics or GitHub issues.

Hiccups

My first working wasm example was a simple print("Hello, world!") which I got working one I built SwiftPM from the main branch. When I then tried to add the JavaScriptKit dependency the runtime let me know that JavaScriptKit requires the reactor ABI, which I promptly built my example with.

Whenever I loaded this latest build (with the reactor ABI) in the browser nothing would happen, even if I included (Int?.none)! to force a fatal error nothing happened. After quite a bit of digging I eventually discovered that JavaScriptKit looks for an exported main function to call which my built wasm didn't have. I ended up solving the issue by adding ["-Xlinker", "--export=main"] to the target's linkerSettings in my package manifest.

Is that meant to be required or might it be something strange about my Swift toolchain? If it's meant to be required as a workaround, it'd be great if that could be added to the docs.

Up-to-date SwiftWASM examples

When trying to resolve my issues I was trying to find example projects to see if I was doing anything differently, but all of the projects that I found hadn't been touched in a few years and either relied on old versions of carton and JavaScriptKit or wouldn't compile. Are there any up-to-date SwiftWASM examples that I missed? If not, I might try updating an existing one for future reference (happy to update whichever example is decided to be the most 'official').

2 Likes

Awesome! Thanks for sharing :slight_smile:

Yes, it's a quirk of WASI reactor ABI and Swift. The workaround would be automatically applied when building with carton, but if you directly use swift build, you need to apply them by yourself.

Anyway, that should be documented somewhere. Which documentation did you refer to when troubleshooting? book.swiftwasm.org?

I have not updated public examples for a while, so your playground is the latest working example :sweat_smile:

1 Like

Yeah, I figured carton might do that automatically. I couldn't get carton working with my build of SwiftPM's main branch (likely due to having main-branch SwiftPM but stable 5.9.2 swiftc?). SwiftPM was expecting an echoLogs key in the output from the plugin but wasn't getting it. The issue was also happening when using plain old swift build from my Xcode toolchain (which also has the issue of having swiftpm 5.9.0 and swiftc 5.9.2 for some weird reason).

Anyway, yeah I think having good instructions for doing things manually would be great. I think it's always important for people to be able to understand what's going on under-the-hood so that if something goes wrong they can work around it.

It could be useful to have a simple repository somewhere basically containing the JS files that I've got in galah's static directory so that people can just drop in their own main.wasm or main.wasm.gz to get things working manually without having to figure out how to transfile the Swift Wasm runtime typescript files from carton or the JavaScriptKit resources from JavaScriptKit. carton would still be recommended, but having this manual alternative would make it easier for other tools to also implement Swift Wasm support.

Alternatively, carton could document how to build the typescript files in entrypoint and how to use them manually (which took me a while to figure out, I had to dig through the code to find the commands used to build the typescript files).

Also, is the Swift Wasm runtime (the swjs set of functions) only contained in Carton? Or is there a non-Carton way to easily integrate the JS-side runtime into other build systems if someone is using React/Svelte etc?

Yeah, I think that's the only real documentation that I found.

Hahah ok :sweat_smile:

Having same errors on the latest 5.10 snapshots

.build/checkouts/swift-syntax/Sources/SwiftCompilerPlugin/CompilerPlugin.swift:103:19: error: cannot find 'dup' in scope
    let inputFD = dup(fileno(stdin))
                  ^~~
.build/checkouts/swift-syntax/Sources/SwiftCompilerPlugin/CompilerPlugin.swift:117:20: error: cannot find 'dup' in scope
    let outputFD = dup(fileno(stdout))
                   ^~~
.build/checkouts/swift-syntax/Sources/SwiftCompilerPlugin/CompilerPlugin.swift:124:11: error: cannot find 'dup2' in scope
    guard dup2(fileno(stderr), fileno(stdout)) >= 0 else {
          ^~~~

stackotter have you had a chance to cross-compile into wasi any project with macros successfully?

1 Like

Cross-compilation for macros is only supported in latest development snapshots of Swift. I won't be cherry-picked to 5.10 and may be cherry-picked to 6.0 only if it becomes stable soon enough.

On latest development snapshots and swift-6 snapshots I'm getting

.build/wasm32-unknown-wasi/debug/JavaScriptKit.build/DerivedSources/resource_bundle_accessor.swift:1:8: error: no such module 'Foundation'
 1 | import Foundation
   |        `- error: no such module 'Foundation'
 2 |
 3 | extension Foundation.Bundle {

unfortunately.

It means that macros is not available for swift-wasm for now, right?

Would be so nice to start using macros for swift-wasm projects

It is available in development snapshots off main only, but not available in 5.10 and 6.0 snapshots.

Sorry for the dumb question

Is it development snapshot of main where fix is available?

I tried swift-wasm-DEVELOPMENT-SNAPSHOT-2024-05-02-a and getting

no such module 'Foundation'

Max_Desiatov could you please point me to the right direction where to get the correct snapshot, I would love to try it :pray:

Yes, that's the correct snapshot that contains the fix for cross-compilation with macros.

I'm not sure what's up with that, could be an issue with the SwiftWasm distribution. Or a bug with SwiftPM resources if you're seeing it with JavaScriptKit exclusively but don't use Foundation in your packages.

@kateinoigakukun have you seen this error before by any chance?

It's hard to diagnose without commands you invoke, but I guess you are using toolchain distribution (not Swift SDK), right? In that case could you try adding --static-swift-stdlib to your swift build command arguments?

1 Like

I'm trying to execute swift build with the following arguments

/Users/imike/Library/Developer/Toolchains/swift-wasm-DEVELOPMENT-SNAPSHOT-2024-05-02-a.xctoolchain/usr/bin/swift build 
    -c debug
    --triple wasm32-unknown-wasi
    -Xswiftc -Xclang-linker
    -Xswiftc -mexec-model=reactor
    -Xlinker -lCoreFoundation
    -Xlinker -licuuc
    -Xlinker -licui18n
    -Xlinker --export=main

adding --static-swift-stdlib seems removes

no such module 'Foundation'

error, but build still fails with

error: link command failed with exit code 1 (use -v to see invocation)
clang: error: linker command failed with exit code 1 (use -v to see invocation)

btw I'm executing rm -rf .build before each build

@kateinoigakukun could you please suggest the right command arguments?

I prepared two clean projects to reproduce the problem

  1. nonmacro project source code
  2. macro project source code

Command that I use to build these projects

/Users/imike/Library/Developer/Toolchains/swift-wasm-DEVELOPMENT-SNAPSHOT-2024-05-02-a.xctoolchain/usr/bin/swift build
    -c debug
    --static-swift-stdlib
    --triple wasm32-unknown-wasi
    -Xswiftc -Xclang-linker
    -Xswiftc -mexec-model=reactor
    -Xlinker -lCoreFoundation
    -Xlinker -licuuc
    -Xlinker -licui18n
    -Xlinker --export-if-defined=__main_argc_argv

nonmacro project builds with no issues

macro project fails:

Fetching https://github.com/apple/swift-syntax.git from cache
Fetching https://github.com/swiftwasm/JavaScriptKit from cache
Fetched https://github.com/swiftwasm/JavaScriptKit from cache (0.90s)
Fetched https://github.com/apple/swift-syntax.git from cache (0.94s)
Computing version for https://github.com/apple/swift-syntax.git
Computed https://github.com/apple/swift-syntax.git at 510.0.2 (0.04s)
Computing version for https://github.com/swiftwasm/JavaScriptKit
Computed https://github.com/swiftwasm/JavaScriptKit at 0.19.2 (0.03s)
Creating working copy for https://github.com/swiftwasm/JavaScriptKit
Working copy of https://github.com/swiftwasm/JavaScriptKit resolved at 0.19.2
Creating working copy for https://github.com/apple/swift-syntax.git
Working copy of https://github.com/apple/swift-syntax.git resolved at 510.0.2
Building for debugging...
warning: Could not read SDKSettings.json for SDK at: /Users/imike/Library/Developer/Toolchains/swift-wasm-DEVELOPMENT-SNAPSHOT-2024-05-02-a.xctoolchain/usr/share/wasi-sysroot
<unknown>:0: warning: libc not found for 'wasm32-unknown-wasi'; C stdlib may be unavailable
warning: Could not read SDKSettings.json for SDK at: /Users/imike/Library/Developer/Toolchains/swift-wasm-DEVELOPMENT-SNAPSHOT-2024-05-02-a.xctoolchain/usr/share/wasi-sysroot
<unknown>:0: warning: libc not found for 'wasm32-unknown-wasi'; C stdlib may be unavailable
<unknown>:0: warning: libc not found for 'wasm32-unknown-wasi'; C stdlib may be unavailable
<unknown>:0: warning: libc not found for 'wasm32-unknown-wasi'; C stdlib may be unavailable
error: link command failed with exit code 1 (use -v to see invocation)
ld: unknown options: --export-if-defined=__main_argc_argv
clang: error: linker command failed with exit code 1 (use -v to see invocation)
[256/261] Linking MyMacroMacros-tool

The snapshot that I'm using is swift-wasm-DEVELOPMENT-SNAPSHOT-2024-05-02-a

@kateinoigakukun could you please take a look :pray:

Interesting. The linker options provided through -Xlinker cannot be used to link macro executables. So you need to specify them at target scope in Package.swift.

        .executableTarget(
            name: "SWasm6App",
            dependencies: [
                "MyMacro",
                .product(name: "JavaScriptKit", package: "JavaScriptKit"),
                .product(name: "JavaScriptBigIntSupport", package: "JavaScriptKit"),
            ],
            linkerSettings: [
                .unsafeFlags(["-Xclang-linker", "-mexec-model=reactor", "-Xlinker", "--export-if-defined=__main_argc_argv"]),
            ]
        ),
1 Like

Thank you @kateinoigakukun this way it works! :partying_face: But it's a bug, right? Since I use different linker options in different situations I can't hardcode them into Package.swift...

Though unsafeFlags condition .when(platforms: [.wasi]) solves it for now

We don't have a way to specify different options for building build-time tools and target executables at this moment, so it might be a missing piece of SwiftPM.

1 Like

I see you've got yours working now too, but yes I have! My Galah programming language has an online interpreter which runs in the browser using Swift Wasm. Currently facing issues with variadic generics but macros are working fine.

1 Like