Cross-compiling with Toolchain files

Development Compiler
1

I'm working on cross-compiling swift using OS X to build a linux-arm
runtime/stdlib, and I've had some success. My change is here:

https://github.com/froody/swift/commit/bc7ca07c1c7a94a8d07acd635c486388640ee2d2

Steps to repro are in cmake/modules/Toolchain-linux-arm.cmake

It's still a work in progress, but I wanted to get some feedback on whether
or not I'm going in the right direction. Toolchain files are the
recommended way for cross-compiling with CMake, but it seems like it
doesn't mesh well with the current system of building multiple mach-o
targets from a single CMake invocation (e.g. OS X, iOS, tvOS and watchOS
built on OS X). I've also had to fight the build-system a lot (using many
--skip-build-foo flags, and two invocations of ./utils/build-script) in
order to avoid building ninja targets that don't exist, and I'm wondering
if there's a better way to do this, i.e. to build llvm/clang/swiftc for OS
X in llvm-macosx-x86_64/swift-macosx-x86_64 build directories, and use that
to build (swiftc?)/stdlib/tests for linux-arm in a swift-linux-armv7 build
directory. Maybe this is a good reason to do
[SR-237] Merge build-script-impl into build-script · Issue #42859 · apple/swift · GitHub first/in parallel?

I also required a change to LLVM:

*hacks to make arm build progress · froody/swift-llvm@ed54c92 · GitHub
<https://github.com/froody/swift-llvm/commit/ed54c92943444999f11918f013cca1dba7892da1&gt;\*

to fix this error:

CMake Error: The inter-target dependency graph contains the following
strongly connected component (cycle):
  "NativeLLVMConfig" of type UTILITY
    depends on "llvm-config" (strong)
  "llvm-config" of type EXECUTABLE
    depends on "NativeLLVMConfig" (strong)

Maybe Chris Bieneman has an idea about this? I seemed to successfully
cross-compile with the "False AND" hack, not sure what's going on.

Anyway, I'd appreciate any feedback on the direction/next steps.

cheers,
Tom

1 Like
2

The main reason we didn't even try to go for the CMake toolchain support is because of the old build/host/target problem. For those who don't know, in old GCC terminology, the build machine is where you build the compiler, the host machine is where you run the compiler, and the target machine is where you run the code that comes out of the compiler.

Right now our CMake project tries to build some binaries for the host machine (the compiler) and some binaries for the target machine (the runtime, standard library, and overlays). On non-Darwin platforms, these are always the same today; for us the host machine is OS X and we build several target standard libraries. CMake really doesn't like this, which is why a CMake-generated Xcode project still can't build for iOS, even though Xcode supports products with different platforms perfectly well.

So from CMake's perspective, we're always building for the host machine, and we pass some funny flags when building the runtime and stdlib. We get away with this because the OS X linker can also link binaries intended for iOS et al.

If you're planning on building a cross-compiler (build = OS X, host = OS X, target = Linux), I think you'd really have the best success by using the same mechanism we use to build multiple stdlibs on Darwin. You'd have to figure out how to trick it into using your linker for just those libraries, but Clang actually does have support for cross-compilation <Cross-compilation using Clang — Clang 18.0.0git documentation; (and CMake should be using Clang to link, I think?) so you should have a chance.

Jordan

···

On Feb 16, 2016, at 13:48 , Tom Birch via swift-dev <swift-dev@swift.org> wrote:

I'm working on cross-compiling swift using OS X to build a linux-arm runtime/stdlib, and I've had some success. My change is here:

https://github.com/froody/swift/commit/bc7ca07c1c7a94a8d07acd635c486388640ee2d2

Steps to repro are in cmake/modules/Toolchain-linux-arm.cmake

It's still a work in progress, but I wanted to get some feedback on whether or not I'm going in the right direction. Toolchain files are the recommended way for cross-compiling with CMake, but it seems like it doesn't mesh well with the current system of building multiple mach-o targets from a single CMake invocation (e.g. OS X, iOS, tvOS and watchOS built on OS X). I've also had to fight the build-system a lot (using many --skip-build-foo flags, and two invocations of ./utils/build-script) in order to avoid building ninja targets that don't exist, and I'm wondering if there's a better way to do this, i.e. to build llvm/clang/swiftc for OS X in llvm-macosx-x86_64/swift-macosx-x86_64 build directories, and use that to build (swiftc?)/stdlib/tests for linux-arm in a swift-linux-armv7 build directory. Maybe this is a good reason to do [SR-237] Merge build-script-impl into build-script · Issue #42859 · apple/swift · GitHub first/in parallel?

I also required a change to LLVM:

hacks to make arm build progress · froody/swift-llvm@ed54c92 · GitHub

to fix this error:

CMake Error: The inter-target dependency graph contains the following strongly connected component (cycle):
  "NativeLLVMConfig" of type UTILITY
    depends on "llvm-config" (strong)
  "llvm-config" of type EXECUTABLE
    depends on "NativeLLVMConfig" (strong)

Maybe Chris Bieneman has an idea about this? I seemed to successfully cross-compile with the "False AND" hack, not sure what's going on.

Anyway, I'd appreciate any feedback on the direction/next steps.

cheers,
Tom
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

3

I agree that CMake really doesn't like building multiple target variants
within a single invocation, which is why I suggested Toolchain files in the
first place. I don't follow your argument that wrapping ld, ar, ranlib,
etc. is preferable to using Toolchain files. The problem is that I'd not
only need to wrap these commands, but also change all the CMake configure
logic, i.e. all the find_package calls, the CMAKE_SYSTEM_NAME tests, and
the check_* calls (luckily there's only one check_symbol_exists call), and
I'm not sure what advantages this has over doing it "the CMake way". Is the
parallelism and the ability to use ninja to rebuild everything without
re-invoking CMake really worth all this extra effort? See
http://thread.gmane.org/gmane.comp.lang.swift.devel/911 for more details.

Just to be clear, are you suggesting building everything (OSX swiftc, OSX
stdlib, Linux stdlib) within a single CMake invocation? Dmitri explained
the reasoning behind doing this for Darwin-based targets (i.e. parallelism,
fast iteration without re-invoking CMake) , but as mentioned above, I don't
think this works well when mixing different toolchains (elf, mach-o) or
headers/libraries (e.g. Linux vs FreeBSD), as you end up having to
re-implement a lot of the CMake configure logic either in shell scripts or
CMake itself. If multiple CMake invocations are ok, why the aversion to
toolchain files?

cheers,
Tom

···

On Tue, Feb 16, 2016 at 10:57 PM Jordan Rose <jordan_rose@apple.com> wrote:

The main reason we didn't even try to go for the CMake toolchain support
is because of the old build/host/target problem. For those who don't know,
in old GCC terminology, the *build* machine is where you build the
compiler, the *host* machine is where you run the compiler, and the
*target* machine is where you run the code that comes out of the compiler.

Right now our CMake project tries to build some binaries for the host
machine (the compiler) and some binaries for the target machine (the
runtime, standard library, and overlays). On non-Darwin platforms, these
are always the same today; for us the host machine is OS X and we build
several target standard libraries. CMake *really* doesn't like this,
which is why a CMake-generated Xcode project still can't build for iOS,
even though Xcode supports products with different platforms perfectly well.

So from CMake's perspective, we're always building for the host machine,
and we pass some funny flags when building the runtime and stdlib. We get
away with this because the OS X linker can also link binaries intended for
iOS et al.

If you're planning on building a cross-compiler (build = OS X, host = OS
X, target = Linux), I think you'd really have the best success by using the
same mechanism we use to build multiple stdlibs on Darwin. You'd have to
figure out how to trick it into using your linker for just those libraries,
but Clang actually does have support for cross-compilation
<Cross-compilation using Clang — Clang 18.0.0git documentation; (and
CMake should be using Clang to link, I think?) so you should have a chance.

Jordan

On Feb 16, 2016, at 13:48 , Tom Birch via swift-dev <swift-dev@swift.org> > wrote:

I'm working on cross-compiling swift using OS X to build a linux-arm
runtime/stdlib, and I've had some success. My change is here:

https://github.com/froody/swift/commit/bc7ca07c1c7a94a8d07acd635c486388640ee2d2

Steps to repro are in cmake/modules/Toolchain-linux-arm.cmake

It's still a work in progress, but I wanted to get some feedback on
whether or not I'm going in the right direction. Toolchain files are the
recommended way for cross-compiling with CMake, but it seems like it
doesn't mesh well with the current system of building multiple mach-o
targets from a single CMake invocation (e.g. OS X, iOS, tvOS and watchOS
built on OS X). I've also had to fight the build-system a lot (using many
--skip-build-foo flags, and two invocations of ./utils/build-script) in
order to avoid building ninja targets that don't exist, and I'm wondering
if there's a better way to do this, i.e. to build llvm/clang/swiftc for OS
X in llvm-macosx-x86_64/swift-macosx-x86_64 build directories, and use that
to build (swiftc?)/stdlib/tests for linux-arm in a swift-linux-armv7 build
directory. Maybe this is a good reason to do
[SR-237] Merge build-script-impl into build-script · Issue #42859 · apple/swift · GitHub first/in parallel?

I also required a change to LLVM:

*hacks to make arm build progress · froody/swift-llvm@ed54c92 · GitHub
<https://github.com/froody/swift-llvm/commit/ed54c92943444999f11918f013cca1dba7892da1&gt;\*

to fix this error:

CMake Error: The inter-target dependency graph contains the following
strongly connected component (cycle):
  "NativeLLVMConfig" of type UTILITY
    depends on "llvm-config" (strong)
  "llvm-config" of type EXECUTABLE
    depends on "NativeLLVMConfig" (strong)

Maybe Chris Bieneman has an idea about this? I seemed to successfully
cross-compile with the "False AND" hack, not sure what's going on.

Anyway, I'd appreciate any feedback on the direction/next steps.

cheers,
Tom

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

4

If multiple CMake invocations are ok, why the aversion to toolchain files?

My main concern is that there end up being two equally-supported ways to build Swift. With LLVM/Clang we had constant small pain trying to keep the autotools and CMake builds in sync. I'd rather not end up in that position for Swift.

(That doesn't mean it might not be the best option. It just means there are serious downsides, and they'd have to be outweighed by the upsides.)

Jordan

···

On Feb 18, 2016, at 9:33 , Tom Birch <froody@gmail.com <mailto:froody@gmail.com>> wrote:

I agree that CMake really doesn't like building multiple target variants within a single invocation, which is why I suggested Toolchain files in the first place. I don't follow your argument that wrapping ld, ar, ranlib, etc. is preferable to using Toolchain files. The problem is that I'd not only need to wrap these commands, but also change all the CMake configure logic, i.e. all the find_package calls, the CMAKE_SYSTEM_NAME tests, and the check_* calls (luckily there's only one check_symbol_exists call), and I'm not sure what advantages this has over doing it "the CMake way". Is the parallelism and the ability to use ninja to rebuild everything without re-invoking CMake really worth all this extra effort? See http://thread.gmane.org/gmane.comp.lang.swift.devel/911 for more details.

Just to be clear, are you suggesting building everything (OSX swiftc, OSX stdlib, Linux stdlib) within a single CMake invocation? Dmitri explained the reasoning behind doing this for Darwin-based targets (i.e. parallelism, fast iteration without re-invoking CMake) , but as mentioned above, I don't think this works well when mixing different toolchains (elf, mach-o) or headers/libraries (e.g. Linux vs FreeBSD), as you end up having to re-implement a lot of the CMake configure logic either in shell scripts or CMake itself. If multiple CMake invocations are ok, why the aversion to toolchain files?

cheers,
Tom

On Tue, Feb 16, 2016 at 10:57 PM Jordan Rose <jordan_rose@apple.com <mailto:jordan_rose@apple.com>> wrote:
The main reason we didn't even try to go for the CMake toolchain support is because of the old build/host/target problem. For those who don't know, in old GCC terminology, the build machine is where you build the compiler, the host machine is where you run the compiler, and the target machine is where you run the code that comes out of the compiler.

Right now our CMake project tries to build some binaries for the host machine (the compiler) and some binaries for the target machine (the runtime, standard library, and overlays). On non-Darwin platforms, these are always the same today; for us the host machine is OS X and we build several target standard libraries. CMake really doesn't like this, which is why a CMake-generated Xcode project still can't build for iOS, even though Xcode supports products with different platforms perfectly well.

So from CMake's perspective, we're always building for the host machine, and we pass some funny flags when building the runtime and stdlib. We get away with this because the OS X linker can also link binaries intended for iOS et al.

If you're planning on building a cross-compiler (build = OS X, host = OS X, target = Linux), I think you'd really have the best success by using the same mechanism we use to build multiple stdlibs on Darwin. You'd have to figure out how to trick it into using your linker for just those libraries, but Clang actually does have support for cross-compilation <Cross-compilation using Clang — Clang 18.0.0git documentation; (and CMake should be using Clang to link, I think?) so you should have a chance.

Jordan

On Feb 16, 2016, at 13:48 , Tom Birch via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

I'm working on cross-compiling swift using OS X to build a linux-arm runtime/stdlib, and I've had some success. My change is here:

https://github.com/froody/swift/commit/bc7ca07c1c7a94a8d07acd635c486388640ee2d2

Steps to repro are in cmake/modules/Toolchain-linux-arm.cmake

It's still a work in progress, but I wanted to get some feedback on whether or not I'm going in the right direction. Toolchain files are the recommended way for cross-compiling with CMake, but it seems like it doesn't mesh well with the current system of building multiple mach-o targets from a single CMake invocation (e.g. OS X, iOS, tvOS and watchOS built on OS X). I've also had to fight the build-system a lot (using many --skip-build-foo flags, and two invocations of ./utils/build-script) in order to avoid building ninja targets that don't exist, and I'm wondering if there's a better way to do this, i.e. to build llvm/clang/swiftc for OS X in llvm-macosx-x86_64/swift-macosx-x86_64 build directories, and use that to build (swiftc?)/stdlib/tests for linux-arm in a swift-linux-armv7 build directory. Maybe this is a good reason to do [SR-237] Merge build-script-impl into build-script · Issue #42859 · apple/swift · GitHub first/in parallel?

I also required a change to LLVM:

hacks to make arm build progress · froody/swift-llvm@ed54c92 · GitHub

to fix this error:

CMake Error: The inter-target dependency graph contains the following strongly connected component (cycle):
  "NativeLLVMConfig" of type UTILITY
    depends on "llvm-config" (strong)
  "llvm-config" of type EXECUTABLE
    depends on "NativeLLVMConfig" (strong)

Maybe Chris Bieneman has an idea about this? I seemed to successfully cross-compile with the "False AND" hack, not sure what's going on.

Anyway, I'd appreciate any feedback on the direction/next steps.

cheers,
Tom

_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev

5

Apologies for being *very* late to this thread and reviving it.

I'm not super familiar with the swift build system or its planned evolution, but I can speak to what we're trying to do in LLVM.

In LLVM & Clang we're taking steps to move away from building for multiple targets in the same CMake invocation. Today the only place we do this is in compiler_rt, and I'll hopefully have that fixed by the end of the summer (assuming I can keep distraction to a minimum).

Our plan is to support an engineering workflow that has a single CMake invocation, but we're going to utilize recursive CMake calls (likely via CMake's ExternalPreject module) to build runtime libraries for one target at a time. This will likely be exposed to the user by allowing them to specify a list of platform-architecture tuples (or GCC/Clang triples), which we'll feed into each of our runtime libraries. I expect to support both Darwin-style cross-compilation (Xcode SDKs), and Linux-style cross-compilation (user-specified sysroots), and I'll either be using CMake Toolchain files, or a similar mechanism to do it.

We've also done extensive work to support Toolchain files for building LLVM and Clang, but we don't currently support Toolchain files for the runtimes in a simple enough way to support building cross-targeted runtimes.

The LLVM community has committed to CMake as our only supported configuration system, and we are committed to making it better at every opportunity. Over the next year (and into the future) we will be working to improve the CMake build systems in all the LLVM runtime libraries (compiler_rt, libcxx, libcxxabi, libunwind,...), with the ultimate goal of being able to support a great cross-compiling experience via CMake for developers and package maintainers alike.

WRT the swift stuff you're trying to do. I might be able to help work out some kinks. The LLVM patch you needed to make leads me to think your toolchain file may not be correct. We use CMake Toolchain files with the LLVM & Clang CMake build systems to cross-compile LLVM libraries and the Clang compiler extensively. The iOS.cmake Toolchain file in LLVM's repo is what we use internally at Apple for most LLVM-based projects that ship on embedded Darwin platforms.

I know this isn't exactly what you're trying to do, but there is a very simple CMake invocation to build LLVM for iOS which works on any Mac with Xcode and an iOS SDK. The CMake invocation is in our documentation here:

http://llvm.org/docs/GettingStarted.html#cross-compiling-llvm

Hope this helps,
-Chris

···

On Feb 18, 2016, at 8:24 PM, Jordan Rose via swift-dev <swift-dev@swift.org> wrote:

If multiple CMake invocations are ok, why the aversion to toolchain files?

My main concern is that there end up being two equally-supported ways to build Swift. With LLVM/Clang we had constant small pain trying to keep the autotools and CMake builds in sync. I'd rather not end up in that position for Swift.

(That doesn't mean it might not be the best option. It just means there are serious downsides, and they'd have to be outweighed by the upsides.)

Jordan

On Feb 18, 2016, at 9:33 , Tom Birch <froody@gmail.com> wrote:

I agree that CMake really doesn't like building multiple target variants within a single invocation, which is why I suggested Toolchain files in the first place. I don't follow your argument that wrapping ld, ar, ranlib, etc. is preferable to using Toolchain files. The problem is that I'd not only need to wrap these commands, but also change all the CMake configure logic, i.e. all the find_package calls, the CMAKE_SYSTEM_NAME tests, and the check_* calls (luckily there's only one check_symbol_exists call), and I'm not sure what advantages this has over doing it "the CMake way". Is the parallelism and the ability to use ninja to rebuild everything without re-invoking CMake really worth all this extra effort? See http://thread.gmane.org/gmane.comp.lang.swift.devel/911 for more details.

Just to be clear, are you suggesting building everything (OSX swiftc, OSX stdlib, Linux stdlib) within a single CMake invocation? Dmitri explained the reasoning behind doing this for Darwin-based targets (i.e. parallelism, fast iteration without re-invoking CMake) , but as mentioned above, I don't think this works well when mixing different toolchains (elf, mach-o) or headers/libraries (e.g. Linux vs FreeBSD), as you end up having to re-implement a lot of the CMake configure logic either in shell scripts or CMake itself. If multiple CMake invocations are ok, why the aversion to toolchain files?

cheers,
Tom

On Tue, Feb 16, 2016 at 10:57 PM Jordan Rose <jordan_rose@apple.com> wrote:
The main reason we didn't even try to go for the CMake toolchain support is because of the old build/host/target problem. For those who don't know, in old GCC terminology, the build machine is where you build the compiler, the host machine is where you run the compiler, and the target machine is where you run the code that comes out of the compiler.

Right now our CMake project tries to build some binaries for the host machine (the compiler) and some binaries for the target machine (the runtime, standard library, and overlays). On non-Darwin platforms, these are always the same today; for us the host machine is OS X and we build several target standard libraries. CMake really doesn't like this, which is why a CMake-generated Xcode project still can't build for iOS, even though Xcode supports products with different platforms perfectly well.

So from CMake's perspective, we're always building for the host machine, and we pass some funny flags when building the runtime and stdlib. We get away with this because the OS X linker can also link binaries intended for iOS et al.

If you're planning on building a cross-compiler (build = OS X, host = OS X, target = Linux), I think you'd really have the best success by using the same mechanism we use to build multiple stdlibs on Darwin. You'd have to figure out how to trick it into using your linker for just those libraries, but Clang actually does have support for cross-compilation (and CMake should be using Clang to link, I think?) so you should have a chance.

Jordan

On Feb 16, 2016, at 13:48 , Tom Birch via swift-dev <swift-dev@swift.org> wrote:

I'm working on cross-compiling swift using OS X to build a linux-arm runtime/stdlib, and I've had some success. My change is here:

https://github.com/froody/swift/commit/bc7ca07c1c7a94a8d07acd635c486388640ee2d2

Steps to repro are in cmake/modules/Toolchain-linux-arm.cmake

It's still a work in progress, but I wanted to get some feedback on whether or not I'm going in the right direction. Toolchain files are the recommended way for cross-compiling with CMake, but it seems like it doesn't mesh well with the current system of building multiple mach-o targets from a single CMake invocation (e.g. OS X, iOS, tvOS and watchOS built on OS X). I've also had to fight the build-system a lot (using many --skip-build-foo flags, and two invocations of ./utils/build-script) in order to avoid building ninja targets that don't exist, and I'm wondering if there's a better way to do this, i.e. to build llvm/clang/swiftc for OS X in llvm-macosx-x86_64/swift-macosx-x86_64 build directories, and use that to build (swiftc?)/stdlib/tests for linux-arm in a swift-linux-armv7 build directory. Maybe this is a good reason to do [SR-237] Merge build-script-impl into build-script · Issue #42859 · apple/swift · GitHub first/in parallel?

I also required a change to LLVM:

hacks to make arm build progress · froody/swift-llvm@ed54c92 · GitHub

to fix this error:

CMake Error: The inter-target dependency graph contains the following strongly connected component (cycle):
  "NativeLLVMConfig" of type UTILITY
    depends on "llvm-config" (strong)
  "llvm-config" of type EXECUTABLE
    depends on "NativeLLVMConfig" (strong)

Maybe Chris Bieneman has an idea about this? I seemed to successfully cross-compile with the "False AND" hack, not sure what's going on.

Anyway, I'd appreciate any feedback on the direction/next steps.

cheers,
Tom

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev