Have you tried using sccache? That should help considerably when working on the build system.
No, I haven't tried it, never heard of it actually. For future reference, I found a section about it just now in the
docs directory after seeing you mentioning it.
Which leads me to another question, are these compiler docs hosted anywhere in a rendered form other than GitHub? Rust has its own Rustc book, which is nicely formatted and supports search, shouldn't we have something similar for Swift so that it's much easier to onboard new contributors to the Swift compiler?
No, I haven't tried it, never heard of it actually. For future reference, I found a section about it just now in the
docsdirectory after seeing you mentioning it.
Hmm, we should probably add a link to this in the Readme.
Which leads me to another question, are these compiler docs hosted anywhere in a rendered form other than GitHub?
No . Earlier, there used to be hosted Doxygen pages for the source code but we don't have that anymore [last time I tried building them a few months back, I ran into sphinx issues and I gave up after a few hours]. I would love to get that set up when I have some spare time. If someone wants to poke around and needs PR review, I can help there.
Rust has its own Rustc book , which is nicely formatted and supports search, shouldn't we have something similar for Swift so that it's much easier to onboard new contributors to the Swift compiler?
A big challenge with comprehensive documentation like that is that it needs to be maintained, much like anything else. Unlike the code, we don't have a compiler that reminds us to update "call-sites", so that needs to be done by hand. For example, I started doing this using named notes, right now there is exactly one of these
[Note: ModuleInterfaceLoader-defer-to-SerializedModuleLoader] . (this is inspired by the notes in GHC)
Plus documenting the internals means that it needs to be changed frequently, compared to the surface language which doesn't change so often. This is reflected in several other documents in
docs/ which are out-of-date at the moment. I agree that having something like that would be extremely valuable for newcomers. IMO we need to take small steps first. That would involve updating the existing documentation, removing what is inapplicable and incorporate external sources about compiler internals, such as blog posts and talks from LLVM Dev.
Now swiftwasm project succeeds to pass quite a few test suite, so we are planning to send some patches to upstream as Max said.
Before sending them, I want to confirm and clarify the direction of supporting WebAssembly.
Implementation design of stdlib
There are some topics to discuss for the implementation of stdlib for WebAssembly.
I know there is some discussion about implementing a subset of stdlib without ICU. But I want to use ICU to support WebAssembly incrementally. I’ve sent some patches to ICU project to build it for WebAssembly.
After these patches applied, it'll get to be able to built for WebAssembly and integrated with Swift stdlib. Actually, I prepared a patched prebuilt ICU library and it works well on WebAssembly with Swift.
Swift stdlib depends on libc but libc can’t be built for WebAssembly because pure WebAssembly doesn't have a system call interface.
wasi-libc is a bytecode alliance project which enables us to build a libc compatible library depending on WASI. As same as I want to use ICU to achieve incremental support, I want to depend on WASI to use wasi-libc also.
We need to provide some polyfills for some missing features like pthread, but swiftwasm project has already implemented them and it works well.
Basically we are focusing to run Swift executable files correctly on WebAssembly runtime, so we want to depend on ICU and WASI to support WebAssembly progressively. I also consider supporting WASI independent target in the future, but for that kind of target, we have to implement a subset of stdlib.
In addition, we want to package wasi-libc and ICU libraries in Swift toolchain of WASM. If a general SDK like wasi-sdk becomes popular in the future, we may use it. But at present, it’s difficult to install wasi-sdk, and unlike the general sysroot, the path to be installed is not fixed. For this reason, we think it is appropriate to distribute them as part of the Swift toolchain at this time.
Implementation design of runtime
As zhuowei said in https://github.com/apple/swift/pull/24684, some of Swift runtime features can’t work on WebAssembly.
Relative Pointer does not work on WASM. See the PR #24684 description for details.
The swiftwasm project has temporarily applied a hacky patch as a solution to this issue. However, the use cases of Relative Pointer have been almost completely identified, and related features also work well.
When merging to the upstream, we plan to implement some kind of abstract pointer type instead of using the RelativePointer type directly. I think that this design should be discussed with the core team more.
swifterror and swiftself don’t work on WASM also. See the PR #24684 description for details.
I have tried to solve this in several ways.
A conversion from non-throws to throws function and a conversion from thin function to thick function is the root cause of this issue.
First, I tried to emit the thunk function at the SIL level. ABI compatibility between function types is checked in the process of converting a typed AST to SIL. For example, conversion from
() -> Int type to
() -> Optional<Int> type requires thunk, but conversion from
()-> Void type to
() throws -> Void type doesn’t require thunk.
This patch https://github.com/swiftwasm/swift/pull/6 enables many cases of function calls to work on WASM. However, there was one problem with this solution.
Thunk functions were created at the SIL level, but in some cases, the functions were rewritten as
thin_to_thick_function instructions by optimization. e.g. CapturePropagation rewrites
partial_apply to a
thin_to_thick_function function. As described above,
thin_to_thick_function instruction outputs function call without thunk and it's an invalid instruction as WASM.
So I implemented IRGen as thin_to_thick_function emits valid LLVM IR for WASM. This change transforms
thin_to_thick_function into function call using LLVM IR level thunk like
partial_apply. Here is the patch PR. https://github.com/swiftwasm/swift/pull/186
These changes make output executable files no longer cause WASM runtime errors in function calls.
WebAssembly doesn't support multi-thread yet, so we run the test suite in a single thread. This makes it difficult to run some test suite that expects to be crashed. Should we ignore these kinds of suite temporarily using ifdef? or is there a better way?
Is this because of a fundamental limitation of the wasm format, or because of missing support in LLVM? The goals of RelativePointer are to (a) reduce metadata size on 64-bit platforms, because it's always 32-bit, and (b) reduce load time overhead by representing references to symbols in the same binary in a way that can be directly encoded in the on-disk image format. Does wasm have an alternative way to achieve this, maybe by using an index into the current object's symbol table, or an address relative to the start of the object file rather than to the PC?
Would it be possible to make it so that
swiftcc functions in LLVM are always lowered with an extra argument and return for the
swifterror argument, regardless of whether they have one or not, so that they still do not need a thunk?
It seems reasonable to expect that a future version of wasm may include some amount of multithreading and atomics support, so "polyfilling" in stub support for pthreads, atomics, and such that really run on a single thread seems like a good approach. Is it practical to provide something that looks enough like multiple kernel threads to fool the tests? If not, disabling them should be fine.
It's due to the design of WebAssembly and LLVM support.
If LLVM supports substruction relocation type on WebAssembly like
X86_64_RELOC_SUBTRACTOR, this issue can be solved easily.
But LLVM doesn't support this relocation type because of the design of WebAssembly.
X86_64_RELOC_SUBTRACTOR are mainly used to generate PIC but WebAssembly doesn't require PIC because it doesn't support dynamic linking. In addition, the memory space also begins at 0, so it's unnecessary to relocate at load time. All absolute addresses can be embedded in wasm binary file directly.
WebAssembly natively achieves the motivation of Relative Pointer without Relative Pointer.
Now we target only wasm32, not including wasm64 because there is no fixed spec for wasm64 and no runtime supporting 64bit.
IMHO we should discuss about wasm32 and wasm64 separately and it's no use discussing non-existing architecture.
As I said above, WebAssembly doesn't need to relocate at load time, so wasm binary format can contain absolute address directly.
And relative pointer has time overhead on dereferencing to calculate real absolute address. On the other hand, if absolute pointers are embedded in binary file directly, there is no overhead at both load time and accessing time.
For these reasons, WebAssembly natively achieves the motivation of Relative Pointer and has a superior runtime performance without Relative Pointer.
I've tried this way also, but it was quite hard to switch the number of arguments for all functions. Here is the WIP patch for this way https://github.com/apple/swift/compare/9466d120ce1b1d79543864938b28bb500f8af4c8...kateinoigakukun:katei/swifterror .
I spent quite a few weeks to solve this issue in this way, but I couldn't list up all function emission code because there is no common helper to emit function in IRGen. (maybe I overlooked)
And I know there is runtime overhead when using thunk, but I don't know that which is more expensive, thunk function or extra argument. What do you think about this?
Update: 2020/02/19 15:21 JST
I think adding extra arguments to all functions costs more than using thunk because the former way has dispatch overheads for every function call and the latter has overhead only when type conversion needed.
No, there isn't.
IMO it's impossible even if there are user level multiple threads because WASI doesn't support signal yet. If we can hook abortion, it may work well.
Thanks for the replies Yuta!
That sounds great, then I agree using an absolute pointer in place of relative is the way to go. However, does WASM support loading and linking multiple wasm files on the client side? How does symbol resolution work in that case? If there's the equivalent of an import table or GOT, it might still be interesting to have something like
RelativeIndirectablePointer, but with an absolute address, to still be able to avoid up-front load time costs for cold metadata structures.
I wouldn't expect an extra added argument and return to have that much overhead, and it seems like it'd be a reasonably tractable peephole optimization for WASM code generators to elide the argument materialization if it were passed an undef value. I would be concerned that, even if you patch up the compiler to thunk instead of assuming the ABIs match up, we still assume that nonthrowing functions are ABI-compatible with throwing functions fairly pervasively through the runtime, and that existing Swift code in the wild may also be relying on that assumption.
If it helps, it's also worth noting that the value in the memory pointed to by the
swifterror argument is generally not significant in the calling convention, only the value at the time the function returns. You could probably avoid materializing an argument altogether and just treat it as undefined on function entry and only materialize an extra return. Another possibility might be, if WASM has a native encoding for alternative returns to represent exception handling, to utilize that to represent error returns and throws instead.
In that case, I would simply disable the tests for now.
WebAssembly MVP doesn't support loading multiple wasm files now and there is no stable ABI for dynamic linking. And also doesn't provide GOT or something related to PIC.
However, LLVM generates virtual static GOT on wasm memory space if the frontend compiler emits IR as PIC. Now, swiftc emits IR as PIC even for wasm, so we have to use something like AbsoluteIndirectPointer to reference through static GOT.
As you said, there is some potential for performance improvement if swiftc emits IR as non-PIC using direct absolute pointer.
Update 2020/02/20 21:38 JST
(I misunderstood what you said, sorry)
If there's the equivalent of an import table or GOT, it might still be interesting to have something like
RelativeIndirectablePointer, but with an absolute address, to still be able to avoid up-front load time costs for cold metadata structures
Do you mean we should use
RelativeIndirectablePointer instead of absolute pointer if there is GOT on WASM?
I see, I basically agree with you at this point. I’ll try this way!
BTW what module should be patched, IRGen or IR lowering module in LLVM? As I said above post, it was quite hard to patch IRGen because I had to modify many places. Do you have any rough plans for this way?
Well, if all wasm code is eventually linked into one object file for deployment, then there shouldn't ever strictly be a need for a GOT (though Swift or LLVM may end up incidentally generating entries because it assumes a symbol might come from a dynamic library in many cases). You might be able to fend off these unnecessary GOT entries in the short term by marking all symbols as
dsolocal in the generated LLVM IR, and changing the logic in Swift IRGen's
getAddrOfLLVMVariableOrGOTEquivalent to always generate direct references when targeting WASM.
If WASM in the future does end up having a dynamic linking story, then we would still probably want to keep the places where we use
RelativeIndirectablePointer as true-const to minimize startup time, but maybe we can avoid solving that problem until the future. A future WASM with dynamic linking support might end up growing full support for PIC too, you never know.
To me, it seems more natural to change the LLVM calling convention lowering for
Thanks for your detailed description! I'll try to eliminate GOT related global variables.
I simply changed the linkage flag and it seems produced object files are linked well :)
OK, I'll take a look around
On WebAssembly runtime, this ABI causes signature mismatch between callee and caller.
Unlike throws and non-throws, this extra generic parameter doesn't have special attribute, so it's hard to lower them as I did for swifterror.
However as far as I investigated, there is only one use case for this ABI, only KeyPath uses it. I've patched to add extra parameter always to match signatures and this patch makes
Do you know other use cases of this optional generic parameter?
In addition, I found extra subscript indices are also passed optionally in KeyPath getter and setter, as same as generic parameter.
Does Swift5 ABI allow additional argument generally? Is this additional argument ABI used in other places?
Adding an extra argument to nongeneric key path entry points is the right thing to do.
Regarding key paths, their weird calling convention is also an issue for optimization, so we may introduce a SIL convention for key path accessors, which would be another marker you could use to lower them to a consistent LLVM function type for wasm:
Thanks! I vote that SIL convention for key path accessors.
Just to confirm, do you mean that extra generic argument and extra indices are only used for KeyPath?
Yeah, I don't know of anywhere else that we expect generic and nongeneric functions to have compatible ABIs. If we add such cases in the future, they should be thunkable for wasm targets.
An update on the current status: if you don't care about binary size, it's more or less usable right now as in this demo. Swift can't run dead code elimination on unused protocol conformances yet, so the minimum produced binary size is ~10M raw and ~1.5M when optimized and compressed with 3rd-party tools, as tracked in swiftwasm/swift#7. Building with SwiftPM works, as long as you don't use Foundation or Dispatch due to a few technical reasons (swiftwasm/swift#592, swiftwasm/swift#658, swiftwasm/swift#647), but I hope that a minimalistic bare-bones subset of Foundation can alleviate that pain in the meantime.
swift test doesn't work yet, but I was able to get a subset of XCTest without expectations, waiters and parallel testing compiling, but not working yet due to a library path issue on Linux. I think I got it working on macOS, but need to verify that with a proper snapshot build. XCTest is one of my top priorities right now, as lack of XCTest blocks people from verifying how well their libraries work when compiled by SwiftWasm.
Overall, I think it's in the state now where any help from the community would be greatly appreciated. Everyone can now install the SwiftWasm toolchain and SDK locally with
swiftenv and check if their package works. I hope to get a GitHub action with the toolchain preinstalled working ASAP so that people can start adding this to their CI. Now that upstream 5.3 has been branched, I'd like to start making 5.3 SwiftWasm snapshots, which hopefully converge to something stable by the time upstream Swift 5.3 toolchain is released. Upstreaming the changes would also tremendously help, as it would reduce the maintenance burden we have due to re-appearing merge conflicts. Technical writing and doing basic "DevRel tasks" is also something that would help a lot. I personally work on this full-time right now, but would be deeply grateful for any help and contributions.
Thanks for reading this
Thanks for the update, this is amazing progress!
This is fantastic! Is there any low-hanging fruit that interested folks could tackle in the area of code size? It seems like the kind of thing where having many eyes on it could make some easy "wins" possible.
This has been previously discussed to some extent in the LTO GSoC thread, but I myself mostly have been focusing on the infrastructure part of SwiftWasm, so I'd rather avoid retelling things I don't know well. I saw @kateinoigakukun have been doing some work and research on this, you can see his posts in that thread. I also know he recently submitted a PR that to me seems to be related to this, but again, I don't have enough expertise in that area to provide more details.
Great work! The topic of dead protocol conformances has come up in our own performance analysis of Swift binaries as a general problem for code size of Swift binaries, so progress on that front would not only benefit wasm but all targets Swift supports.