Embedded operating system in Swift - Custom standard library / using the Builtin module

Development Standard Library
#1

A while ago I found an interesting blog post about writing an operating system in Rust.

I'm curious, what it would take to implement something similar using Swift?
I did read this post:
Creating a new Platform, SDK, and toolchain, resources?

however I did not find improvements in this area on any roadmap so far.

Are there any interesting compiler flags that would let me cross compile for any LLVM supported architecture without creating a completely new toolchain from scratch? I know I can avoid linking the standard library, but that's not very useful when even integers are part of it. Is it possible for non standard library code to access the builtin types without building the compiler somehow?

Other issues I can think of are lack of inline assembly etc.

I really don't want to build Swift from source, last time I tried I ran out of SSD space and my MacBook crashed.

i.e. what would it take to at least draw "Hello world!" to the screen without an operating system?

edit:
As a starting point it would be interesting to know how to print hello world just generating an object file, without the Swift module; Linking with libc/darwin and writing to stdout is not enough because I have no way of expressing arrays of integers.

And that's a problem still even if I somehow have access to the builtin module, because how can I implement an Int8 array without pointers?
I would probably have to add these first, right?

But to do anything I first need builtin types; I assume it would be pretty simple to just compile for whatever architecture I need, or add assembly files :slight_smile:
I searched everywhere but I couldn't find anything, and the Swift source code is massive

ok I figured out this part, can I get Builtin to show up in VS Code?

I think what I'm essentially trying to do is write a custom less OS dependent "standard library" from scratch, I should probably move this to a different category

edit 2:
Another issue, I'm stuck with top level code but it would probably be nice to have a main() function; I could probably do so using clang, but I'm not sure if it's going to cause issues when Swift is in top level code mode; can swiftc generate normal Swift?

Where can I even find documentation on this? :confused:

3 Likes
Swift with no standard library
#2

I think that you might find GitHub - compnerd/uswift: μSwift[Core] fairly interesting then. It is meant to be a stripped down standard library for embedded environments (needs more work of course).

7 Likes
#3

Thank you, that is really interesting, I'm happy there is at least some interest in making Swift more portable :slight_smile:

That's probably more than enough to start.

I noticed that the bare metal target support patch has not been merged, and I am currently not able to build the entire Swift compiler. Could this be compiled manually using clang?
Or alternatively is there a way to build Swift without taking up 90gb of space :smile:

1 Like
#4

I need to find some time to look through that patch again and fix the test failure that it was triggering. The best you can do is build without debug info, but I don't know how to do that with build-script unfortunately. I think that might bring it down significantly.

3 Likes
#5

Sounds like a good hackathon project.

Assuming you could compile swift itself (even with some corners cut), you can do quite a lot without standard library.

The following should print "Hello, World!" without OS and with some minimal assumptions (video memory starts there, mode is B&W, with that by that dimensions, etc)

Hello, World! app 
import Swift

func setByte(_ byte: UInt8, at address: UnsafeMutablePointer<UInt8>) {
    address.pointee = byte
}

func setBytes(_  bytes: [UInt8], at address: UnsafeMutablePointer<UInt8>) {
    for (index, byte) in bytes.enumerated() {
        address[index] = byte
    }
}

private let videoMemoryAddress = 0xC000
private let screenBaseAddress = UnsafeMutablePointer<UInt8>(bitPattern: videoMemoryAddress)!
private let stride = 80     // assuming 640 pixel wide B&W screen
private let colCount = 80   // assuming 640 pixel wide B&W screen and 8x8 letters
private let rowCount = 25   // assuming 8x8 letters

func drawLetter(_ letter: Int, column: Int, row: Int) {
    guard column >= 0 && column < colCount && row >= 0 && row < rowCount else {
        return
    }
    var row = row
    for byte in letters[letter]! {
        screenBaseAddress[row * stride + column] = byte
        row += 1
    }
}

func drawLetters(_ letters: [CChar], column: Int, row: Int) {
    var column = column
    var row = row
    for letter in letters {
        if letter != 0 {
            drawLetter(Int(letter), column: column, row: row)
            column += 1
            if column >= colCount {
                column = 0
                row += 1
                if row >= rowCount {
                    row = 0
                }
            }
        }
    }
}

let letters: [Int : [UInt8]] = [
    0x20: [ // space
        0b00000000,
        0b00000000,
        0b00000000,
        0b00000000,
        0b00000000,
        0b00000000,
        0b00000000,
        0b00000000,
    ],
    0x21: [ // !
        0b00001000,
        0b00001000,
        0b00001000,
        0b00001000,
        0b00000000,
        0b00000000,
        0b00001000,
        0b00000000,
    ],
    0x2C: [ // ,
        0b00000000,
        0b00000000,
        0b00000000,
        0b00000000,
        0b00011000,
        0b00001000,
        0b00010000,
        0b00000000,
    ],
    0x48: [ // H
        0b00100010,
        0b00100010,
        0b00100010,
        0b00111110,
        0b00100010,
        0b00100010,
        0b00100010,
        0b00000000,
    ],
    0x57: [ // W
        0b00100010,
        0b00100010,
        0b00100010,
        0b00101010,
        0b00101010,
        0b00101010,
        0b00010100,
        0b00000000,
    ],
    0x6C: [ // d
        0b00000010,
        0b00000010,
        0b00011010,
        0b00100110,
        0b00100010,
        0b00100010,
        0b00011110,
        0b00000000,
    ],
    0x65: [ // e
        0b00000000,
        0b00000000,
        0b00011100,
        0b00100010,
        0b00111110,
        0b00100000,
        0b00011100,
        0b00000000,
    ],
    0x6C: [ // l
        0b00011000,
        0b00001000,
        0b00001000,
        0b00001000,
        0b00001000,
        0b00001000,
        0b00011100,
        0b00000000,
    ],
    0x6F: [ // o
        0b00000000,
        0b00000000,
        0b00011100,
        0b00100010,
        0b00100010,
        0b00100010,
        0b00011100,
        0b00000000,
    ],
    0x72: [ // r
        0b00000000,
        0b00000000,
        0b00101100,
        0b00110010,
        0b00100000,
        0b00100000,
        0b00100000,
        0b00000000,
    ],
]

drawLetters(Array("Hello, World".utf8CString), column: 0, row: 0)

Note that it uses string literal and arrays but it would be relatively easy to not use those (perhaps by typing (0x48, 0x65, 0x6C, 0x6C, 0x6F, 0x2C, 0x20, 0x57, 0x6F, 0x72, 0x6C, 0x6C, 0x21) instead of "Hello, World!" and emulating (fixed) arrays with tuples. Minimally for this task you need bytes, ints, addition operation, tuples of bytes and ability to cast an integer (video memory address) to a tuples of bytes. As an alternative to tuple of bytes you'd need UnsafeMutablePointer of bytes (as shown in the above sample). Maybe you'll be able removing "import Swift", emulate byte with enumeration of 256 cases, emulate Int as 2, 4 or 8 bytes and make your own primitive operations like +, but I didn't explore that. Good luck with your project.

7 Likes
#6

i have always wondered why the swift compiler needs 90 GB of disk space to compile (90? last time i tried it was ~80 GB), because this was always an obstacle for me when i wanted to test out a change to the toolchain. what exactly is so disk-intensive about building the toolchain? would be great if some compiler developers could explain what it is using all that space for, and if there is a way to build a stripped-down toolchain just for the purposes of testing out changes to the language.

3 Likes
#7

Until I'm able to compile an unknown target capable Swift compiler, I could try linking with the library on macOS

From what I understand, Apple by default only allows using the system provided library:
error: -static-stdlib is no longer supported for Apple platforms
How can I provide swiftc with an alternative standard library and statically link with it?

I'm not sure, I ran out of space and macOS stopped working. 90 was about how much I had before I started compiling

1 Like
#8

Hmm.
I successfully compiled the patched toolchain, and that did give me swiftc, however I did so with these settings

utils/build-script --skip-build-benchmarks \
  --skip-ios --skip-watchos --skip-tvos --swift-darwin-supported-archs "$(uname -m)" \
   --release --swift-disable-dead-stripping --bootstrapping=off

It was surprisingly fast and didn't take much space, but I'm not sure if this is a functional compiler because it fails to compile a hello world, saying it could not find -lSystem

Not sure how to make a toolchain out of this. It would be really nice to have documentation, the getting started guide says nothing about this.

1 Like
#9

update:

Something I noticed though, is that the compiled swiftc still does not recognise none as a valid OS target :eyes:
error: unknown target 'aarch64-unknown-none'

Unfortunately, I do do not have the C++ skills required to understand swiftc so there isn't much I can do.
I can definitely do things with Swift code, and I don't mind spending a lot of time on this, but I will not get the toolchain to work. I do not have the experience required to work on a large compiler, especially one with as little documentation on even the build process.

I can only describe the documentation and build process as a huge roadblock for anyone interested in the language.

This might be obvious, but without the standard library, what is the limitation preventing swiftc from supporting all targets available to llvm?
I don't like mentioning Rust all the time, I don't really like it that much as a language, but it has very understandable documentation on how to add new targets. (and a lot of detailed documentation in general)

#10

There was this project called BetaOS that was supposed to be an x86 operating system written in as much Swift as possible. GitHub - admkopec/BetaOS: An operating system written in Swift it was a cool project but no longer compiles on modern macOS. I could never get it to run in an emulator and the developer has no plans to continue it. It seemed like a cool project. I would love to see someone else go at it.

1 Like
#11

This is also something I'm missing with the Swift compiler. If you look at Clang which is usually shipped with Swift or is required in order to use the Swift toolchain, the compiler supports many target CPU architectures.

clang -print-targets
  Registered Targets:
    aarch64    - AArch64 (little endian)
    aarch64_32 - AArch64 (little endian ILP32)
    aarch64_be - AArch64 (big endian)
    arm        - ARM
    arm64      - ARM64 (little endian)
    arm64_32   - ARM64 (little endian ILP32)
    armeb      - ARM (big endian)
    thumb      - Thumb
    thumbeb    - Thumb (big endian)
    wasm32     - WebAssembly 32-bit
    wasm64     - WebAssembly 64-bit
    x86        - 32-bit X86: Pentium-Pro and above
    x86-64     - 64-bit X86: EM64T and AMD64

However, the swiftc compiler seems to be like old GCC you have only one target (you can probably have more if you add some and recompile). What I really like with Clang is that there is a separation between gode generation and OS support (which are libraries). I think that Swift should have the same possibilities out of the box, supporting a myriad of CPU targets out of the box (the precompiled versions).

The arm-none-eabi targets are usually the starting point for OS/embedded development. From there it's the library support and hopefully the Swift project could make the library modular enough in order to support different levels of OS support.

2 Likes
#12

I think it would be nice to have a swiftc flag, something like -custom-swift-stdlib, and maybe allow a new target for packages (like .executable() and .library()) called .standardLibrary(), which would have access to the Builtin module, since as far as I know swiftc flags disallow an external package from being imported. that would make it so much easier to work on these.

Then in the dependencies: [] array you would just put the custom standard library, and it would automatically switch to it.
There would still need to be a way to easily choose the target for swift packages, including cross compilation for multiple targets.

But it would be enough to just have swiftc capable compiling to llvm targets, though not as easy to work with as the packages.

That's interesting, I think I even found this project a while ago, though didn't look that much into it because it wasn't being worked on for a while now.

I hope it would be possible to get away with just Swift and Assembly :slight_smile:
It would also be helpful if the new experimental macros feature could be used to implement an inline assembly #asm macro somehow

1 Like
#13

Broadly speaking, assembly should mostly not be necessary. You would still like to have it for some specific critical sections, but everything else ought to be possible via a matured Builtin module (and for aforementioned critical sections, I would prefer to use asm files rather than propagate the million-dollar mistake that is asm() in C compilers).

3 Likes
#14

You get assembly language for "free" these days. Clang offers assembler file support out of the box. The LLVM toolchain offers a GCC style inline assembly syntax. Not sure if Swift supports this right now but it is just a matter of exposing this in the Swift compiler, the LLVM compiler takes care of the rest.

#15

Sure, you can put inline assembly in a C header file and use it from Swift today. My point is that it shouldn't be necessary to do this, and further you shouldn't want to, because C inline assembly has been pretty disastrous historically (and, to be clear, I say this as someone who has shipped at least thousands of lines of it, and much more non-inline assembly than that).

#16

Yes, you get it for "free", as long as you ignore all the additional costs associated with parsing, validating, piping to the backend, piping diagnostics back through the stack, and then dealing with late errors poorly. clang has a large amount of code in the frontend to deal with this handling. While LLVM does have the integrated assembler, it is mostly meant to avoid the need for an additional tool to re-implement the assembly handling rather than make this all "free". The integration is anything but free (speaking with experience as I've had to fix both the inline assembly handling as well as the validation that goes along with it).

This is absolutely incorrect. The swift frontend is absolutely a cross-compiler. The code generation is mostly split out from the OS support (there could be better code separation through abstraction but that is mostly an implementation detail and an engineering trade-off).

I think that the problem is that you are misunderstanding a fundamental design point decision in Swift: everything is a library. That is, you can target a free-standing environment, as long as you are willing to give up a few things, such as Int, String, Array, etc. These are all library level implementations. There are no fundamental types in Swift like there are in C. As a result, you must have a library for the fundamental types. To support those types you need dynamic memory allocation which currently uses the global system memory allocation routines to enable interoperability and thus requires the presence of a libc, which will inherently drag along the assumptions on the host environment.

The triple is mapped to compile time checks (e.g. os(...)), which is what my previously mentioned patch adds support for. I've just not had time to go through and address the fallout of those changes. The limited need standard library is what uSwift is trying to address.

2 Likes
#17

Speaking of that, I really liked what Rust did in order to open up for adding more targets without the need of adding compile time checks.

https://book.avr-rust.com/005.1-the-target-specification-json-file.html

Basically a JSON file describing the target.

#18

That isn't so simple with Swift. The point is that the compiler checks the code irrespective of the configuration. That is,

#if os(Windows)
let x = 32
#elseif os(None)
let y = 64
#else
let z = 128
#endif

has all three branches checked when compiling irrespective of the target. As a result, the compiler has to understand the OS configuration and map that to the appropriate types. A good example of why this matters is the example of Windows where C types such as CLong and CLongLong are mapped differently than on LP64 environments. IMO Swift is more careful about accidentally breaking a non-compiling configuration, and that is worth the extra complexity in adding support for a new target.

More importantly to your point of the JSON file, I'd argue that Swift has an even more elegant solution to that problem: no need for the configuration at all!

The configuration that the file defines is:

  1. flags (e.g. -fPIC, -fuse-ld=, -nostdlib, -Xlinker ---eh-frame-hdr, -mcpu=..., -lgcc, etc
  2. a subset of the GCC spec files to translate certain arguments (e.g. default values for -o)

With Swift, you don't need to configure most of this as we pull that information directly from clang and LLVM, and the only thing that you need to specify is -target, at which point this entire file will boil down to:

{ "llvm-target": "..." }
3 Likes
#19

How do I access a register without assembly though? Do I need to create a separate assembly file? wouldn't that be slower than accessing them inline? As far as I know Swift has no way of directly accessing anything

So if this works, I could just set arm-none-eabi as a target, arm7tdmi as the cpu, and compile for a Nintendo GBA without any json files? :slight_smile:

#20

Something interesting I found today, not exactly embedded but related to cross compilation:

I'm a bit confused how it's supposed to work, what about the standard library?
I can't exactly test it because someone decided to disable the -static-stdlib flag on a certain operating system, even when compiling for a different target it seems :upside_down_face:

It seems like swiftc is only used for ir here?
Could something like this be done with a more recent version of Swift?