Creating a new Platform, SDK, and toolchain, resources?

For SwiftWasm we don't use the upstream script to build an Xcode toolchain, we assemble it in our separate toolchain distribution script from tarballs produced in separate CI runs. And it's only done on CI once a day if a new build is ready to be distributed.

Regardless, you don't need to a toolchain package to test what you've just built, unless you're specifically interested in how Xcode and SwiftPM integration works. Building a toolchain .pkg every time you make a change to test is detrimental to the development iteration time in the edit/build/debug loop.

In most of the cases you'll use swiftc directly from the build directory to test something, and it's relatively rare that you need to do that manually. Most of the bugfixing workflows look like this:

  1. add a test to the test suite to reproduce some behavior you're interested in;
  2. make sure it fails with lit.py invocation;
  3. make an attempt to fix the test in the codebase, add logging output and/or connect with lldb to debug if needed;
  4. run ninja to incrementally rebuild what you've changed;
  5. re-run your test with lit.py to verify the fix, use verbosity flags to get more output;
  6. if the test still fails, goto 3.

None of these steps require building the full toolchain package installable in Xcode.

I was hoping to keep SwiftPM working at all times but I suppose I donā€™t need Xcode support so youā€™re right, creating the toolchain is kind of irrelevant right now.

So then I guess my next big step is to build an armv8-a variant of the swift standard library.

This is where I got overwhelmed before.

The smaller steps seems to be:
Add the platform to llvm.
Add the platform to swift tools core package.
Add the platform to SwiftPM.
Add the armv8-a arch to SwiftPM.
Find and update the arch limitations for the platform.
Update the makefiles to support the new target.
Add the includes for platform specific C stuff (pthread, etcā€¦)
And Iā€™m sure thereā€™s a ton Iā€™m missing.

Seems like Iā€™ll have to peck away at these 1 by 1 with builds being broken for long periods of time.

Do these steps at least seem somewhat right to you?

IDK, maybe steps for every platform would be different. Here's how I would proceed, and this somewhat overlaps with the plan you've proposed, but I'd postpone SwiftPM support for as long as possible. Runtime and stdlib are much more crucial. Also, I'm not sure what you mean by "Swift tools core package", this seems too broad as a definition. Here's how I would've started with SwiftWasm (or any other new sufficiently different platform) knowing what I know now:

  1. Find a C++ cross-compilation toolchain that targets your platform. Prepare a sysroot environment with libc for it, or find tools that do that for you. Ideally it should be based on clang/LLVM and allow creating simple projects with CMake, separate from the whole build system of the Swift toolchain. This will be your playground. Check out how CMake cross-compilation toolchain is set up, you'll need that toolchain file for things to work. This means your playground C++ cross-compilation toolchain will be as close as possible to how Swift runtime bits will be compiled for your target platform, as Swift runtime is written in C++. Your playground project will basically try to replicate the Swift runtime separately from the whole Swift toolchain and SDK. This will allow you to test things in isolation.

  2. Your goal is to get the runtime source code cross-compiled and working. Without it, stdlib won't work. Pick the smallest bit of Swift runtime that you can isolate in your playground. Numeric.cpp looks like that to me, and it doesn't include much stuff, only a couple of header files. Don't hesitate to plainly copy those headers into your playground, even if it's LLVM code, like the ArrayRef data structure. Basic LLVM infra code is frequently copied around, a copy of it even exists in the Swift git repository to be included in the Swift runtime itself. You don't need to cross-compile the whole LLVM in your playground, just these small infrastructural bits. Get rid of whatever is unnecessary to avoid distractions, and isolate the smallest amount of code you can cross-compile that still overlaps with the Swift runtime code.

  3. If you can get Numeric.cpp cross-compiled with what it includes, note the changes that you had to make, if any. Given how small that file is, it's unlikely you'll have to make fundamental changes, but just completing this exercise you'll learn a lot about Swift runtime and how its source code is structured. Understand if these changes are really necessary for your port to work, submit them upstream as PR if so. Then you can proceed with the next smallest and isolated runtime source file, maybe Heap.cpp. I'm not a runtime expert, maybe there are better candidate files, or maybe it's hard to find isolated pieces and you'll have to get major pieces of runtime cross-compiled at once in your playground. After all, it's a trial and error learning process.

  4. You may need some testing harness in your playground. Something in C++ that loads this newly cross-compiled isolated runtime code and tries to do something useful with it, like calling demangleSymbolAsString with a known mangled symbol and examining how that works. Or trying to work with RelativeIndirectPointer. After you're successful with that, you can get metadata and heap allocations working.

  5. Proceed with cross-compiling more and more runtime code to your target platform. Maybe your platform isn't that different from the existing platforms and the whole runtime will compile at once just by giving it a correct CMake cross-compilation toolchain file, then you won't need to dissect small runtime bits in a separate playground. But I think a separate playground Swift runtime project is still going to help a lot. You won't be overwhelmed with the rest of the Swift codebase and will be studying things in isolation.

  6. Try to submit code needed for the port to work as PRs upstream as soon as possible. The earlier you do this, the more feedback you'll get. This will also help you understand the contribution workflow. After you get the runtime working, you'll have to add support to the Swift driver for your platform. This will likely mean the legacy C++ driver and the new Swift driver, depending on how things are currently set up.

  7. With the working driver you can start trying to compile stdlib and running the stdlib test suite for your platform. Also, check out the uSwift project for a minimal stdlib codebase to work with. You'll need a good way to run tests, lit.cfg may need some modification to support this. You'll probably have to create a preset for your platform either at this stage or a bit later. This will help you make build-script work, and it'll be easier to manage all your cross-compilation options.

I think when you reach this point, things will be a lot more clear in terms of subsequent steps. By this time you'll probably understand what works and what doesn't on your platform, like Unicode and multi-threading. Based on that, you can decide whether you want to port concurrency bits, Dispatch, Foundation, and XCTest. This may also have an impact on how you add SwiftPM support.

8 Likes

This approach is brilliant :grin:
I would have wasted a lot of time before it occurred to me to do it this way.

With this approach Iā€™m a lot less deterred about the GBA, but I think the Nintendo Switch is probably still a better 1st platform.

Thank you so much for taking the time to write this up. I really appreciate it!

2 Likes

I have a working Swift toolchain for Nintendo Switch (targeting libnx) using swift-embedded as a base. It works but it's a giant hack / requires its custom SPM fork / frontend script etc... The biggest hack is that I shoehorned newlib into clang and called it a toolchain.

Everything in the language itself works but without Foundation you are obviously quite limited. Libnx works out of the box thanks to the great C integration Swift has. I even tried nanovg for hardware accelerated vector graphics, and it works great.

I never bothered to port libdispatch and Foundation so I never took the time to cleanup the port and release it. I can share it here if you want, as a base for your own research or if you want to continue from what I did. I don't want to discourage you, I know the fun is to make it yourself!

7 Likes

That'd be awesome :)

Please do! That would be super helpful.
libnx is the library I was going to go with so thatā€™s perfect.

I donā€™t mind if itā€™s hacked together. Having a reference would still be super helpful :sunglasses:

Alright here it is: klepto Ā· GitHub. So far it's only tested to work on Ubuntu 20.04. I just dumped everything on GitHub with a dummy commit message a while back and moved on. It does not work without SwiftPM - you can call swiftc manually but you have a ton of devkitpro options to give.

devkitpro must be installed in /opt/devkitpro due to how GYB files work in the build script. It's the same limitation as why we need(ed) a glibc path patch in the Raspberry Pi port of Swift.

Here's a quick rundown of what's in every repo / why I needed to fork that specific project:

The only part that's not included is a libicu build for libnx. I did not bother adding it to the build script, but you can make it yourself quite easily by cloning the libicu fork of Apple on GitHub and hitting make with the correct cross-compilation flags.

Speaking of metadata, this is one of the things I didn't try at the time. I don't know if the necessary metadata sections are copied to the NRO file, and if they are I don't know if the runtime can access them.

Image introspection (for stack traces) is explicitely disabled since its current ELF implementation uses dlopen and we don't have it. I did not bother implementing an NRO implementation.

As for what was to come next, I tried to port Foundation but saw that it depended on libdispatch. So I tried to port libdispatch but I saw that it was heavily dependent on Linux features such as uevents and futexes.

The Switch kernel has some similar mechanisms so a port could be feasible but I could not be bothered and I lost interest. My wacky newlib-into-clang toolchain also triggers an obscure clang bug with __atomic in newlib's stdatomic, making it impossible to build libdispatch in its current state. I found an issue on the clang issue tracker about that exact same error but I can't find it again, sorry.

I hope that's everything, good luck! Please let me know if/when you have something that works best that what I have, I would love to keep making Switch homebrews in Swift this time! And don't hesitate to ask if you have questions on the Switch homebrew ecosystem / on why I made specific choices on my port attempt.

12 Likes

Thanks for sharing. I wasnā€™t expecting to get such a big head start!
These will be a very helpfully reference :grin:

Iā€™ll see if I can get a ā€œhello worldā€ running on my Switch and then do a clean fork and implement only the essentials. Maybe move your SwiftPM stuff into one of those new SPM plugins, which I havenā€™t looked at yet but it seems like a good candidate for creating a switch app from a binary.

Dispatch and Foundation are non-optional for my engine so Iā€™ll definitely be tackling those too.

And my efforts will definitely be open source so Iā€™ll make a post when I have something usable.

4 Likes

You're welcome!

I considered using a plugin but then I thought about having swift run run the homebrew using nxlink and I don't think a plugin can currently do that. I don't even know if you can add "post-build" commands for elf2nro that needs to happen after linking.

romfs generation is definitely possible using a plugin though, I experimented with them a bit and resources processing is super simple and works great.