Swift for bare-metal/RTOS based microcontroller


(Andy Lau) #1

Hi folks!

This is my first thread and sorry it seems a little long. I made a title for each part, you can pick what you are interested in : )

There are few discussion about using Swift on microcontrollers. Maybe most of you guys here are pure software engineers. I’m an electronic engineer, I’m eager to use such a modern language in my work(I mean bare-metal/RTOS based microcontroller, nothing to do with Raspberry Pi, that's a totally different thing). Just image, you can truly leave the Apple system and using Swift language for IOT hardware, like arduino, mbed or MicroPython do, but in a more modern language. That's what I plan to do.

In the past half year, I was trying really hard to make this dream come true, below is all the related information.

Hardware and traditional toolchain:

To achieve my purpose, I chose the ARM Cortex-M7 chip for my experiment. The performance of the chip is excellent. The LLVM backend has good support for it. And I can add extra ROM and SDRAM to my hardware if
Swift consumes too much sources. Once the experiment succeed, I can manage to produce a bunch of them in a short time. I use Arm Gcc toolchina https://developer.arm.com/open-source/gnu-toolchain/gnu-rm/downloads for their headers, libc, libm, libgcc, libstdc++ etc. It matchs the ARM chip very well.

Modifying to the compiler(Swift 4.2 right now):

Thanks to the talented structure of LLVM. It's not hard to add my own BareMetal toolchina triple(thumbv7em--none-eabi for now) to the Swift Compiler. I need to thanks @Brian_Gesiak here. His blog modocache.io gave me a deep insight to the swift compiler! I also add "-function-sections" and "-fdata-sections" to the compiler which is very important to reduce the final size of a static linked object file.

Bare-metal or RTOS:

Refer to this project: https://github.com/spevans/swift-project1 (Thanks to @spevans). It is a truly bare-metal swift program on the x86 machine. It’s really an awesome project. Simon Evans implemented all the lower stuff he need to run his swift program(a swift kernel) in bare-metal. Most of the lower stuff are related to memory management which the runtime relied on. In my practice, I want to do this easier. I choose a RTOS called Zephyr to be my lower kernel, so I don’t need to care about memory management. In addition, this Zephyr is POSIX compatible, so it's easier to port thread related stuff later.

Cross compiling the stdlib:

After I add my own triple to the compiler, it's very smooth to compile the whole stdlib.

Cross compiling the Runtime:

The clang already support triple thumbv7em--none-eabi, so it's not hard to compile most of the runtime files(Just did some miner change for the incompatible places). At first, I thought all the *.mm files in the runtime and stubs directory are Objective-C related. Since I won't use OC at all, I excluded them when I compiling. But then I found even with my pure easy swift program, it might emit calls to the functions in those .mm file. I don't know why. So here comes the question:

1. When compiling the Runtime and stubs files, can I exclude all the *.mm file in the runtime and stubs directory if I only use pure swift on my hardware? If so, how can I prohibit the call to those function in the *.mm files?

The biggest problem now, libicu, UNICODE stuff:

When compiling the stubs/UnicodeNormalization.cpp, I found it is heavily depend on the libicu, which is another huge project http://site.icu-project.org/download. After some research, seems no one tried to cross compile the libicu to a bare-metal microcontroller, it is heavily relied on system call itself. At first, I thought this might affect the swift String type a lot, if it's impossible to cross compile the libicu. I may just have to implement my own C/C++ like String type. But I was wrong, all the swift stdlib are heavily relied on libicu. It's so frustrating. Many basic type such as Dictionary would emit runtime call to the libicu : (

I don’t know shall I implement my whole microstdlib?(like @carlos4242 is doing) First, that's a huge of work. Second, I want my porting remain the same as the official Swift. Here comes my another question.

2. Do you guys have any good idea on how can I handle the libicu stuff?

Any related information would be helpful. Thanks very much!
Andy


(Lars Sonchocky-Helldorf) #2

Hi @madandylau, have you seen these posts on the topic:

and


(Andy Lau) #3

Hi @IOOI, thanks for your kind reply!

I’ve noticed those topics before. @JetForMe seemed very interested in this topic also. I’ll say hi here : )

What they were discussing is how to reduce the features of Swift language so it can fit the microcontroller. What I am trying to do is keeping the language features as much as possible. So I cannot avoid the swift runtime(written in C++) and stdlib(written in swift). These two parts constitute libswiftCore.

The background of how a swift application runs
@spevans gave an explanation here https://github.com/spevans/swift-project1/blob/master/doc/kstdlib.md about his swift kernel on a bare-metal x86 machine. Normally, the libswiftCore need support from libc++, libc, libicu, and some OS functions. He implemented most things in his tiny klibc library. It’s just like a tiny OS + libc + libc++. And he stubbed out many features he doesn't need, such as libicu support and thread support.

What I’m trying to do is similar. But I don’t need to implement everything myself. There are already good choices. The Arm GCC toolchain has provided libc and libstdc++. The Zephyr RTOS has provided memory management which is needed by swift runtime.

baremetalframwork

Why not Raspberry Pi
Maybe you will ask why I don’t just use Raspberry Pi, they already support Swift. I think any electronic engineer could tell you the answer. It’s too complicated on both hardware and the Linux OS. Too much things are out of control to make a stable electronic product. Especially for the Linux OS, in contrast to the RTOS, you can even implement you own in less than 1000 line of codes. Even I choose the Zephyr RTOS as my kernel, it’s not particularly more complicated than bare-metal.

The difficulties I've met
Since I had already added the triple thumbv7em—none-eabi to my swift compiler. It’s no problem to emit object file which can run on the cortex-m7 chip directly. The RTOS just provide very fundamental functions to the upper layer. What I’m struggling with is the different library dependency and compatibility problem. It's really a headache. If I want to support most of the function in stdlib, I think I need some way to provide the libicu functions. That's how most of stdlib types depend on(String type for example).

If someone here can give me some hint, I'll appreciate that a lot. Thanks you guys!
Andy


(Simon Evans) #4

There are a couple of possible solutions for the libICU issue although none of them are particularly easy:

  1. As you mention, try and cross compile it. Most of the rest of the swift toolchain can be cross compiled and given that new versions of ICU are required for support in swift-corelibs-foundation, support for cross compilation of libICU would be welcomed so any work on this would be good. Hopefully there will be other people who also want to get this working as it will be useful for all platforms.

  2. Keep an eye on https://bugs.swift.org/browse/SR-9432 and https://bugs.swift.org/browse/SR-9423 which both seem to be about enhancing the stdlib to just use the data from libicu which should remove the cross-compilation issue for you while still giving full unicode compatibility.

  3. If you don't really need unicode support you could just treat the UTF-8 strings as bytes and try and modify the stdlib to remove ICU support entirely and eliminate normalisation etc. Note that this would require quite a lot of changes to the String code and also bear in mind that maintaining external patches to stdlib quite be quite a lot of work as there is a lot of code changes which move things around etc.

1 & 2 are probably the best way to go.


(Andy Lau) #5

Hi @spevans,

I'm so happy to get your reply here : ) I've learnt a lot from your project. It's really awesome. It gave me confidence to start this project.

At present, I just stub out the libICU related functions in swift runtime like you did. The compiling and linking seems right. After the linking procedure, I can get an ELF executable file. Then I need to use arm-none-eabi-objcopy to generate a binary file for the MCU.

Here comes a new problem. If my swift code is easy enough(just contain basic types or classes, such as array), everything works fine. I can generate the final binary(the size is around 100KB) and download it to my MCU. But if I use array.append() or print() function in my swift code, the final ELF file would contain a section named .got. And the final binary size would be unreasonable huge(around 400MB). I'm still working on this problem. If you have any advice on this, that would be great : )

As an electronic engineer, I have to confront much unfamiliar area. Thanks for your great work again, it really saved a lot of my time : )


(Simon Evans) #6

Just a few random thoughts:

After you have linked your executable but before you use arm-none-eabi-objcopy you need to check the executable is statically linked. You can also use readelf -aW to check the size of the sections in the ELF file to see if any are really large.

If your working file was only around 100KB then it wouldn't be linking in the runtime and stdlib. This is contained in a file called libswiftCore.a and will be around 10MB but you will see error messages about undefined references if it is not being linked in.

The 400MB file could be caused by some code or data being at a large offset, ie if you have you code at absolute location 0 but your data is at absolute location 256MB then objcopy will have to pad the gap in between with zero's to get the data at the right location.

The Program Header section in the readelf output should show the VirtAddr and PhysAddr address that the sections are being loaded at. The MemSiz field will show if one of these sections is excessively large.

The .got table is the Global Offset Table used for storing absolute addresses. If you haven't seen it before this document http://www.skyfree.org/linux/references/ELF_Format.pdf about the ELF format is worth
reading.

Hope some of that helps!


(Andy Lau) #7

Since I compiled the swift stdlib with -function-sections and -data-sections(I added these two options to my swift compiler). The final ELF would just contain those codes it need. So it might be much smaller than the whole libswiftCore.a

Your information are very useful. I think there's something wrong with my linking script and I'm trying to figure it out. I'll give updated information here if I make some progress.

Thanks for your help!


Runtime error in the BareMetal environment for microcontroller
(Andy Lau) #8

It's a long time since my last post. During these time, I solved the linking problem. It's caused by the default linker script provided by Zephyr.

The default linker script puts some sections generated by the swift compiler in a wrong address, that's why the final binary was that huge. After a little fix, the whole procedure works very well now! The final binary is around 1.1MB.

The good news, most swift codes can run on my cortex-m7 board now! I tried a lot of code snippets, most of them work very well(Different types, Functions, Classes, Structures, Protocols, etc.). But the print function still didn't work, after some time debugging. I found that when the program goes into the line if case let streamableObject as TextOutputStreamable = value in _print_unlocked function in OutputStream.swift. The program would crash. Thanks to the Zephyr RTOS, it provides me some fault information. It's related to wrong memory access.

Till now, I found some similar problems. They all crashed when run into the swift_conformsToProtocolImpl function in ProtocolConformance.cpp. I've added a new topic on this problem.

Thanks for all your help @spevans, I plan to make my own board in recent months. I must send you one after I finished. :grin:


(Tenkai Kariya) #9

Hi Andy,

This is super interesting! I myself am an embedded systems engineer and I am looking to try and run swift on an ARM Cortex M, (ideally an M4, but I'd start at an M7 to see how it all works)

Do you have any documentation on your configuration that you used to get this to work? I'd love to be able to replicate your experiments!

EDIT: I actually read through your first post, and it seems like you outline all your technologies right there. I am going to try to replicate this on a nucleo board! Do you have your linker scripts uploaded to github anywhere? I'd love to check em out!


(Andy Lau) #10

Happy to see you are interested in this : ) It's really hard for embedded engineer to touch modern programming language. Even it's 2019 right now, most of us can only use C for programming.

At present, there isn't something like a one-touch script. I modified a lot in the swift compiler source code especially the runtime, then I have to compile the files one by one. More than that, I also did some minor change in the Zephyr project(The base RTOS) and GNU Arm Embedded Toolchain(Compile the RTOS and do the final linking).

Since the final bin file is more than 2MB, it's hard to run the program on a traditional board(The max ROM is normally 2MB). I'm designing a PCB board based on NXP RT1052 this month. It would contain an external NOR Flash from which I can get 16MB ROM.

After that, I plan to make a simple one-touch IDE like Arduino did. It would do the whole process automatically. Please stay tuned!


(Tenkai Kariya) #11

Awesome, I don't expect that it is a one touch script, for sure, but if you had some documentation on how you accomplish this compilation process, along with the changes you made to the Zephyr project and your toolchain, I might be able to help you create a set of scripts to compile a project.

I use PlatformIO and VSCode as my IDE, and it is all script based and I could create a custom platform to put all this together on.

I also have some friends that are very interested in this as well, including some who work on Swift at Apple. My idea is to start an open source project on this so we can have the most exposure and support to make something real happen, with more support for more processors and smaller libraries so that we are able to create binaries that are small enough to run on a processor with less than 256k memory.


(Andy Lau) #12

Open source all the code is planned. Merge all the modification into the main swift project is the best choice. But I have to choose the right time. In my plan, I need to do following things one by one

  1. Make one specific PCB board, I will find the balance between the performance and cost. I think the final price would be similar to MicroPython pyboard, but the performance is much much higher than it. Both hardware and programming language.

  2. Make a simple one-touch IDE. I don't want any complex function. I just want to make it as simple as possible. It can compile the swift code, the RTOS, and link them at background. Then it's OK. At this point, I will release all the executable file which used for compiling.

  3. Write swift wrapper for the low level drivers, find and fix the compiler bugs, make it stable. Then most swift programmers can use this board. It's just another choice beyond Arduino and MicroPython

  4. Release all the source code, port the whole system to other microcontroller platforms.

During step 3-4, anyone interested in this project could join in. And I will write detailed documents at that time.

At present, all the different toolchains just work in my Linux virtual machine. Plenty of work need to be done. I will give updated information here once I get some progress.