What are best practices to write a Linux Software on macOS

Server
1

One of the issues I am having with Swift is that it currently seems to be non-trivial to figure out whether a given code snippet will work on Linux or not. As I understand the open source Foundation isn't yet complete. This wouldn't be too big of a problem but the documentation isn't very helpful (when looking at a class in Foundation it's not obvious whether this has yet been implemented in the open source version).

It seems that while SPM and the compiler has ways to detect whether something will work with a targeted OS X version but there's no way to target Linux. Is this correct? This just seems to me like Swift server is a second class citizen... Is there plans to improve on this?

It seems in the near future Swift might be as convenient as C++ in that regard (as long as you only use the standard library and no OS specific stuff you'll probably be fine), but I was hoping Swift could do better in this area since it seems most of the building blocks necessary are already there?

What I am trying to do is this: build a server application but use my mac as a dev machine. So while I want it to compile and run on macOS my primary target is Linux. What is the best practice of doing this?

I obviously could wait for CI to tell me, but it doesn't seem to be super convenient. I would really like to be in a place where if my stuff compiles and works on macOS there's a very high chance it will also compile and work on Linux.

6 Likes
2

No, that's not correct. The compiler only sees availability. The issue is that Apple's Foundation doesn't include Linux availability, so the only way to see whether your code will compile is to actually build on Linux. This is fairly easy if you can do it locally or using a CI system like GitHub Actions, which is free for public repos. The only way to know if something will work is to run your test suite on Linux to exercise Foundation for Linux.

Others probably have better solutions, but one way to build for Linux locally is to set up a local Docker instance using the Swift images and share your code with that image. That should allow you to do local Mac and Linux builds at roughly the same time. You can also just virtualize a whole Linux instance with something like Parallels.

3

This is already the case today for the Swift Standard Library. Note that Foundation and Dispatch are not considered part of the Swift Standard library and instead separate frameworks.

If you use platform specific features you can just use docker to build and run swift on linux locally on your mac. After you have downloaded and installed docker you can just run the following command:

docker run \
    --rm -it \
    --volume "$(pwd)/:/src" \
    --workdir "/src/" \
    swift:5.7-focal

and afterwards swift build or swift run name-of-exectuable to build or run your code on ubuntu linux.

Cross Compilation from macOS to Linux will be possible with this pitch: Pitch: Cross-Compilation Destination Bundles. You will then no longer need docker to build your code for linux. If you want to run it you will still need a Linux VM after all.

More server specific guides are available on Swift.org - Swift on Server Guides

3 Likes
4

I’d also consider multipass: https://multipass.run/

We use it with a good result.

5

You can already use the Swift compiler (and package manager) as a cross compiler. What you need is a destination.json file, then you can build your package this way:

swift build --destination destination.json

The tricky bit with using the macOS Swift compiler as a cross compiler for Linux is that (at least at the moment) the macOS compiler is built with the Swift driver enabled, while Swift for Linux is still built with the C++ driver. This causes the compiler to complain if you try to link cross-compiled modules with the Swift standard library for Linux. A way to work around this is to build a Swift toolchain (on the Mac) that is built with the same options as the native Linux one (i.e. building with the --skip-early-swift-driver turned on) and install that (including the corresponding destination.json) under /Library/Developer/Toolchains. Then you can use, e.g.:

xcrun -toolchain my.swift.toolchain.identifier swift build --destination /Library/Developer/Toolchains/swift-5.7.3-linux.xctoolchain/destination.json

As already mentioned, the simplest way at the moment is to just use the native toolchain for your version of Linux within a docker container (see the available Docker images on https://www.swift.org/download/).

1 Like
6

I use the following:

Pull the latest with:
% docker pull swift

Compile and test your package in the latest image on your machine. Once complete the container is removed:

% docker run -it \
  --rm \
  --mount src="$(pwd)",target=/foo,type=bind \
  swift \
  /usr/bin/swift test --package-path /foo
7

Thanks all for the answers! I think for now I will use docker (and give multipass a shot).

To summarize: it seems the biggest problem is that Foundation isn't yet fully ported. This will probably be a non-issue in the near future and so I just have to live with the current state until then. This shouldn't be a big deal. Cross-compilation will improve the situation even more and it makes me happy that this is in the works!

However, I still wonder (and I might misunderstand something here) whether it wouldn't be possible to do even better.

For context: I believe that in a developer cycle common problems should be caught very early and less common problems can be caught later:

  • Typos and syntax mistakes are common (at least for me), so the compiler should catch those immediately -- optimally the IDE will catch it while I type
  • Very severe bugs that just break everything I should be able to catch by just running a smoke test -- and it shouldn't matter whether I run these tests on Linux or macOS.
  • More subtle bugs should be caught on any OS but might take longer to be found by the testing system -- so they might only appear in CI or even the nightly tests
  • Bugs caused due to the compiler or standard library having different behavior on Linux vs macOS should optimally be very rare

So I'd like to get my product into a state where somebody can just clone the repository, change some code, and then run swift test. Afterwards there should be a high probability that the code changes didn't introduce new problems (obviously this mostly requires good tests and only a small part of the problem is the tooling).

I guess one last question I have about this:

Say I have a class Foo defined like this:

#if os(macOS)
class Foo {
// defition
}

then I have a function defined like this:

func bar() {
  let f = Foo()
  // do something with f
}

I would expect this to work on macOS but this won't compile on Linux because Foo won't exist. But it feels like the compiler should be able to tell me something like "this won't compile on all the platforms you're targeting". So basically what I'd like to do is to annotate Foo like this:

@available(macOS 13, *)
class Foo {}

and then tell Swift build that I want code that can be compiled on Linux. So if Foo is used without being guarded by #available compilation would fail.

It seems this is possible across macOS platforms. Is this also possible for Linux?

8

It probably won’t be fully ported (if not done by some community effort), but there actually is a new implementation of Foundation planned which will then be “equal” on all Swift platforms see this blog entry and this forums entry (note that more according future topics for discussion are announced there) and also being discussed in other topics. (The usage of the new implementation might then require some adjustments in your code, e.g. because of the planned modularization.)

9

If your project is a SwiftPM project you can use Visual Studio Code dev containers to verify a Linux build? These allow you to edit, build and debug projects inside a Docker container. You can find out more here vscode-swift/remote-dev.md at main · swift-server/vscode-swift · GitHub

5 Likes
10

I've used exactly this technique in a couple of projects. I mean, yeah - it would be really nice if there was a better option, but at least being able to invoke and work the code inside a docker container through VSCode is an incredibly useful workaround.

11

Unfortunately right now, the only way to know is to just build it on Linux . It is unfortunate that Apple doesn't just include Linux availability in their documentations.

3 Likes
12

The tricky bit with using the macOS Swift compiler as a cross compiler for Linux is that (at least at the moment) the macOS compiler is built with the Swift driver enabled, while Swift for Linux is still built with the C++ driver. This causes the compiler to complain if you try to link cross-compiled modules with the Swift standard library for Linux. A way to work around this is to build a Swift toolchain (on the Mac) that is built with the same options as the native Linux one (i.e. building with the --skip-early-swift-driver turned on)

Much of what you say is correct, but this part isn't. Both the official linux and mac toolchains use the driver written in Swift. You can check this for yourself with the linux toolchain:

> ls -l swift-5.7.3-RELEASE-ubuntu20.04/usr/bin/swift-driver
-rwxr-xr-x 1 butta butta 38773896 Jan 15 19:55 swift-5.7.3-RELEASE-ubuntu20.04/usr/bin/swift-driver
> ./swift-5.7.3-RELEASE-ubuntu20.04/usr/bin/swift --help
...
Welcome to Swift!

Subcommands:

  swift build      Build Swift packages
  swift package    Create and work on packages
  swift run        Run a program from a package
  swift test       Run package tests
  swift repl       Experiment with Swift code interactively

  Use `swift --help` for descriptions of available options and flags.

  Use `swift help <subcommand>` for more information about a subcommand.
> ./swift-5.7.3-RELEASE-ubuntu20.04/usr/bin/swift --help -disallow-use-new-driver
...
SEE ALSO - PACKAGE MANAGER COMMANDS:
        "swift build" Build sources into binary products
        "swift package" Perform operations on Swift packages
        "swift run" Build and run an executable product
        "swift test" Build and run tests

Note the different help text based on which driver is invoked.

Whatever compiler errors you're getting when cross-compiling are unrelated to which compiler driver is used, as both mac/linux toolchains should be using the new driver written in Swift. --skip-early-swift-driver is only used when building the linux toolchain because the linux build doesn't require a prebuilt Swift toolchain, so it uses the legacy C++ Driver to build everything up till the new driver written in Swift, then builds, installs, and uses that new Swift driver.

3 Likes
13

So I followed the main advice here and I am now using a docker container which works reasonably well.

However, it seems Foundation on Linux and macOS are incompatible. Is it possible to use the same version of Foundation on macOS and Linux? I understand the one on Linux is a re-implementation in Swift and C -- so it should be possible to also use on my mac?

14

There's a longer-term solution that's being aimed directly at this - reworking foundation itself: Swift.org - The Future of Foundation.

15

I am aware of this and IIUC on Linux the new foundation version is shipped. Is there a way I can use the same version on macOS?

It's just very annoying to have subtle differences between the two

16

No, that’s not the new foundation. For practical purposes the foundation you have on Linux is what you see is what you get until the new one comes.

17

Hi Markus!
Great to see you here, and sorry I've not chimed in earlier but I see the forums folks have been rather helpful/informative so far :slight_smile:

I can clarify the Foundation situation a bit. (And we absolutely share the pain you feel

The foundation team actively working on open sourcing a "Foundation implemented in Swift" which then will be the same across all platforms. That plan was shared here in some more detail: What’s next for Foundation and all I can add to that is that the work is still actively underway :crossed_fingers: It's been a long time coming thing and we're very happy to finally properly solve this annoyance that you've rightfully pointed out here.

Until then you sadly cannot "pick" the open source one while developing on macos.

For what it's worth for server side projects we often simply default to not using Foundation "as a whole" but e.g. specifically importing just Date, or JSONEncoder etc. Which may make this adventure also easier for you. I'm more than happy to help with specific trouble you hit and we sometimes have foundation-free solutions to things, feel free to ping me about any needs you hit.

7 Likes
18

Thank you! I think I got everything working for now. I am looking forward to the time when I can the same implementation of Foundation on macOS and Linux, but for now I can make it work. One thing that really helps is that Swift compilation times are great (at least for someone who is used to C++ :joy:).

For reference (or if someone else in the future is interested in this), I used docker-compose to test locally and with GH actions. This works quite well, though for GH actions I couldn't figure out how to get SPM caching to work (I copy&pasted the official example but it doesn't work). But for now I don't worry as the whole job takes like 5 minutes and I have more free minutes than I can spend anyways (also debugging CI issues is like the least fun thing I can think of right now).

I'll share my config here since I assume it's not an uncommon setup for someone doing Swift on the server: this setup works for applications that use GRPC and PostgreSQL.

Dockerfile:

version: "3.8"
services:
  base: &base
    build: .
    image: swarm:latest
    volumes:
      - "..:/src"
      - "/etc/passwd:/etc/passwd:ro"
    working_dir: /src
  bash:
    <<: *base
    command: bash
  build:
    <<: *base
    command: swift build

  docs:
    <<: *base
    command: swift package generate-documentation

  db:
    image: postgres:15
    restart: always
    ports:
      - "5432:5432"
    environment:
      POSTGRES_PASSWORD: test

  test:
    <<: *base
    environment:
      - PSQL_HOST=db
      - PSQL_PORT=5432
      - PSQL_USER=postgres
      - PSQL_PWD=test
    command: swift test
    depends_on: 
      - db

docker-compose.yml:

version: "3.8"
services:
  base: &base
    build: .
    image: swarm:latest
    volumes:
      - "..:/src"
      - "/etc/passwd:/etc/passwd:ro"
    working_dir: /src
  bash:
    <<: *base
    command: bash
  build:
    <<: *base
    command: swift build

  docs:
    <<: *base
    command: swift package generate-documentation

  db:
    image: postgres:15
    restart: always
    ports:
      - "5432:5432"
    environment:
      POSTGRES_PASSWORD: test

  test:
    <<: *base
    environment:
      - PSQL_HOST=db
      - PSQL_PORT=5432
      - PSQL_USER=postgres
      - PSQL_PWD=test
    command: swift test
    depends_on: 
      - db

One pitfall I had to figure out is that the test needs to wait for PostgreSQL to come up. I just retry every 2 seconds for 60 seconds which until now has worked every time. It will read the connection information from the environment variables.

I would've preferred to start PostgreSQL from within the test and I spent 2 days trying to figure this out, but PostgreSQL people really don't want you to do that it seems (I could go on a long rant here how PostgreSQL seems to hate best practices and common sense and I would like to point out that it took me 5 minutes looking at their internal testing scripts to find a race condition that would make your tests flaky if you reuse them, but that's not relevant in this context).

3 Likes