[RFC] Toolchain based build process

ddunbar · 2016-06-01T21:18:29.000Z

Hi all,

The current build process for the overall Swift project (i.e., the compiler + associated projects like Foundation, XCTest, and SwiftPM) relies on each project having dependencies on the built artifacts of previously built projects. Those dependencies are currently communicated to each project in the chain through an ad hoc set of arguments to the individual project's build process. This has been painful to maintain, and makes it hard to reason about the environment that each of the associated projects are building within.

Instead, I would like to move towards what I have been calling a "toolchain-based" build process.

In this model:

1. The entire build process will be organized as a sequential, incremental construction of a complete toolchain.
  - Each individual project will build and copy its products into a staging area.
2. The build script will always create a composed toolchain.
  - It will start with an empty toolchain, and merge in the content from each project as it builds.
  - This will use rsync & hard-links to avoid needing to stay fast.
3. Each individual project build will just use the composed toolchain to build.
  - This will replace the grab bag of options we use to communicate between the projects.
4. At the end of the build, we will have constructed a complete toolchain which can be installed (or used with Xcode).

Aside from simplifying the overall build process, this has a couple very nice upsides:

1. At the end of the build, the user has a complete toolchain. They can install it, or use it as they would a distributed snapshot. This is very beneficial for people who are only building the Swift project to get a more recent version of the compiler.

2. Each individual project can be built using the "official" build process with a downloaded snapshot (assuming it is of a new enough version). This is very beneficial for easing contribution to projects like Foundation, XCTest, and SwiftPM which are pure Swift and have fast build times, but which currently build in Swift CI using a very different process from the snapshot-based workflow.

Concrete details:

1. I do not plan to change the actual install process in the short term. The actual install process used today relies on the CMake-based install process for some projects (most importantly Swift) and isn't suitable for use in this fashion (where incremental development speed is of high importance). Instead, my plan is to teach the build script itself how to assemble the staging area for each individual project, with the long term goal of using that for the official install process instead of the CMake-based process.

2. My plan is that the build script would only support building one "primary product" (i.e. toolchain). That product may itself be a complete cross compiler toolchain with support for multiple platforms, but the expectation is that users would invoke the build script multiple times if building multiple toolchains. However, to support Canadian Cross [1] build scenarios the build script may need to manage the construction of two products, the primary toolchain and the intermediate (cross compiling) toolchain used to build the artifacts in the primary toolchain.

3. As a concrete example of a problem this solves, SwiftPM currently doesn't test its `swift-test` product in Swift CI, because that product requires using all of the toolchain stack (swift, Foundation, and XCTest), but the product itself is only intended to be used as part of a concrete toolchain. As such, it doesn't know how to operate correctly when given the piecemeal build products for each of those projects, and teaching it to do so would be very cumbersome for us to maintain.

Feedback welcome!

- Daniel

[1] Cross compiler - Wikipedia

compnerd · 2016-06-03T20:14:46.000Z

Hi all,

The current build process for the overall Swift project (i.e., the
compiler + associated projects like Foundation, XCTest, and SwiftPM) relies
on each project having dependencies on the built artifacts of previously
built projects. Those dependencies are currently communicated to each
project in the chain through an ad hoc set of arguments to the individual
project's build process. This has been painful to maintain, and makes it
hard to reason about the environment that each of the associated projects
are building within.

Instead, I would like to move towards what I have been calling a
"toolchain-based" build process.

In this model:

1. The entire build process will be organized as a sequential, incremental
construction of a complete toolchain.
        - Each individual project will build and copy its products into a
staging area.
2. The build script will always create a composed toolchain.
        - It will start with an empty toolchain, and merge in the content
from each project as it builds.
        - This will use rsync & hard-links to avoid needing to stay fast.
3. Each individual project build will just use the composed toolchain to
build.
        - This will replace the grab bag of options we use to communicate
between the projects.
4. At the end of the build, we will have constructed a complete toolchain
which can be installed (or used with Xcode).

Aside from simplifying the overall build process, this has a couple very
nice upsides:

1. At the end of the build, the user has a complete toolchain. They can
install it, or use it as they would a distributed snapshot. This is very
beneficial for people who are only building the Swift project to get a more
recent version of the compiler.

2. Each individual project can be built using the "official" build process
with a downloaded snapshot (assuming it is of a new enough version). This
is very beneficial for easing contribution to projects like Foundation,
XCTest, and SwiftPM which are pure Swift and have fast build times, but
which currently build in Swift CI using a very different process from the
snapshot-based workflow.

Concrete details:

1. I do not plan to change the actual install process in the short term.
The actual install process used today relies on the CMake-based install
process for some projects (most importantly Swift) and isn't suitable for
use in this fashion (where incremental development speed is of high
importance). Instead, my plan is to teach the build script itself how to
assemble the staging area for each individual project, with the long term
goal of using that for the official install process instead of the
CMake-based process.

2. My plan is that the build script would only support building one
"primary product" (i.e. toolchain). That product may itself be a complete
cross compiler toolchain with support for multiple platforms, but the
expectation is that users would invoke the build script multiple times if
building multiple toolchains. However, to support Canadian Cross [1] build
scenarios the build script may need to manage the construction of two
products, the primary toolchain and the intermediate (cross compiling)
toolchain used to build the artifacts in the primary toolchain.

3. As a concrete example of a problem this solves, SwiftPM currently
doesn't test its `swift-test` product in Swift CI, because that product
requires using all of the toolchain stack (swift, Foundation, and XCTest),
but the product itself is only intended to be used as part of a concrete
toolchain. As such, it doesn't know how to operate correctly when given the
piecemeal build products for each of those projects, and teaching it to do
so would be very cumbersome for us to maintain.

Feedback welcome!

Overall, I think that this would be a great change.

I am however slightly confused on why the build script needs to be
concerned about candian cross at all if it can be taught how to use a
specified toolchain.

Because these (standard at least as per auto tools) terms can be confusing:

[1] build - the platform where things are being built
[2] host - the platform where the generated binaries will be run
[3] target - the platform where the binaries generated by the compiler on
the host platform will run

It should be possible to shift the burden on the user. They would be
responsible for the first stage of the canadian cross (to build a toolchain
capable of generating binaries for the host platform). Then invoke the
build script a second time with the toolchain they just built to
cross-compile the cross-compiling toolchain.

Or did I completely misunderstand your detail point 2 and this is the
expectation?

- Daniel

···

On Wed, Jun 1, 2016 at 2:18 PM, Daniel Dunbar via swift-dev < swift-dev@swift.org> wrote:

[1] Cross compiler - Wikipedia

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

--
Saleem Abdulrasool
compnerd (at) compnerd (dot) org

ddunbar · 2016-06-03T20:31:40.000Z

Hi all,

The current build process for the overall Swift project (i.e., the compiler + associated projects like Foundation, XCTest, and SwiftPM) relies on each project having dependencies on the built artifacts of previously built projects. Those dependencies are currently communicated to each project in the chain through an ad hoc set of arguments to the individual project's build process. This has been painful to maintain, and makes it hard to reason about the environment that each of the associated projects are building within.

Instead, I would like to move towards what I have been calling a "toolchain-based" build process.

In this model:

1. The entire build process will be organized as a sequential, incremental construction of a complete toolchain.
        - Each individual project will build and copy its products into a staging area.
2. The build script will always create a composed toolchain.
        - It will start with an empty toolchain, and merge in the content from each project as it builds.
        - This will use rsync & hard-links to avoid needing to stay fast.
3. Each individual project build will just use the composed toolchain to build.
        - This will replace the grab bag of options we use to communicate between the projects.
4. At the end of the build, we will have constructed a complete toolchain which can be installed (or used with Xcode).

Aside from simplifying the overall build process, this has a couple very nice upsides:

1. At the end of the build, the user has a complete toolchain. They can install it, or use it as they would a distributed snapshot. This is very beneficial for people who are only building the Swift project to get a more recent version of the compiler.

2. Each individual project can be built using the "official" build process with a downloaded snapshot (assuming it is of a new enough version). This is very beneficial for easing contribution to projects like Foundation, XCTest, and SwiftPM which are pure Swift and have fast build times, but which currently build in Swift CI using a very different process from the snapshot-based workflow.

Concrete details:

1. I do not plan to change the actual install process in the short term. The actual install process used today relies on the CMake-based install process for some projects (most importantly Swift) and isn't suitable for use in this fashion (where incremental development speed is of high importance). Instead, my plan is to teach the build script itself how to assemble the staging area for each individual project, with the long term goal of using that for the official install process instead of the CMake-based process.

2. My plan is that the build script would only support building one "primary product" (i.e. toolchain). That product may itself be a complete cross compiler toolchain with support for multiple platforms, but the expectation is that users would invoke the build script multiple times if building multiple toolchains. However, to support Canadian Cross [1] build scenarios the build script may need to manage the construction of two products, the primary toolchain and the intermediate (cross compiling) toolchain used to build the artifacts in the primary toolchain.

3. As a concrete example of a problem this solves, SwiftPM currently doesn't test its `swift-test` product in Swift CI, because that product requires using all of the toolchain stack (swift, Foundation, and XCTest), but the product itself is only intended to be used as part of a concrete toolchain. As such, it doesn't know how to operate correctly when given the piecemeal build products for each of those projects, and teaching it to do so would be very cumbersome for us to maintain.

Feedback welcome!

Overall, I think that this would be a great change.

I am however slightly confused on why the build script needs to be concerned about candian cross at all if it can be taught how to use a specified toolchain.

Because these (standard at least as per auto tools) terms can be confusing:

[1] build - the platform where things are being built
[2] host - the platform where the generated binaries will be run
[3] target - the platform where the binaries generated by the compiler on the host platform will run

It should be possible to shift the burden on the user.

Why would we _want_ to shift the burden to the user?

From the user's perspective, they want to end up with a toolchain that compiles `Host -> Target`... I see it as an implementation detail (of the `build-script`) that producing what the user wants requires us to build the intermediate `Build -> Target` toolchain.

They would be responsible for the first stage of the canadian cross (to build a toolchain capable of generating binaries for the host platform). Then invoke the build script a second time with the toolchain they just built to cross-compile the cross-compiling toolchain.

The tricky part of the canadian cross situation isn't the toolchain capable of generating binaries for the host, it is the toolchain capable of generating binaries for the target, but which needs to run on the build host. This is needed when you need to produce runtime libraries that will execute on the Target, but you are simultaneously trying to construct a cross compiler for a Host which you cannot actually execute. This is when you need the extra `Build -> Target` toolchain, and is the situation I was referring to.

- Daniel

Or did I completely misunderstand your detail point 2 and this is the expectation?

- Daniel

···

On Jun 3, 2016, at 1:14 PM, Saleem Abdulrasool <compnerd@compnerd.org> wrote:
On Wed, Jun 1, 2016 at 2:18 PM, Daniel Dunbar via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

- Daniel

[1] Cross compiler - Wikipedia

_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev

--
Saleem Abdulrasool
compnerd (at) compnerd (dot) org

Karl · 2016-06-10T09:38:58.000Z

We can’t separate building for ‘Build’ and building for an arbitrary ‘Host’. For those who don’t know, this is how cross-compiling works with Swift (this isn’t off-topic, I’m going to bring it back firmly to this proposal):

Currently, when you install the compiler, the standard library is in (prefix)/lib/swift/(Target)/(arch). On OSX, we cross-compile the standard library for the iOS targets, so you’ll see subfolders for ‘macosx’, ‘iphoneos’, etc in there. All you need to cross-compile in Swift is the standard library; swiftmodules don’t need ‘ar’, so libraries don’t even need any binutils to cross-compile, which is refreshing. A cross-linker *is* unfortunately still required for executables (come on, LLD!).

Grab the standard library from a linux machine and try:
echo 'print("hello, world")' | ./swiftc —target=x86-unknown-linux-gnu -resource-dir {PATH_TO_LINUX_STDLIB} -emit-module -

So when we’re cross-compiling the swift portions of the standard-library, we’re using the swift compiler we just made for Build, telling it to use Host’s lib/swift directory, and there it will find the correct subdirectory for Target. My native compiler for my OSX Build machine knows not to look for the Linux host’s own Target in swift-macosx-x86_64/lib/swift, but in swift-linux-armv7/lib/swift.

···

—

Okay, so that’s out of the way.

There is an actual problem with how we build/configure the target libraries (Foundation, XCTest and libdispatch). We currently configure them like every other product (for each host), instead of for each target. A consequence of this is, for example, that we won’t try to build Foundation for cross-compile targets such as Android. What we should do is this:

For each host:
-> Build cmark/llvm/swift
-> For each target of this host:
-> -> Build Foundation/XCTest/libdispatch
-> Build swiftpm

When you look at it like this, from a tools // target-libraries perspective (swiftpm is an exception; it’s a host tool, but it depends on Foundation), I came up with a different idea (possibly, I’m not sure), that might also help the dependency issue, although I don’t know the specifics of it: I would suggest that after building swift, we install it, then build the target libraries and install them as we do so.

That would give us a stable standard-library (lib/swift) location which we could treat like an installed system. So when Foundation builds (for each Target now), it sets -resource-dir to the installed Host copy of swift, and the Build swift compiler can find Target’s standard library (yeah, I know). It installs itself in there. Then XCTest comes along, it sets the same -resource-dir, so now it can find Foundation just with ‘import Foundation’ and nothing else. In other words: every product after the swift compiler can assume its dependencies are like they’ve been installed on the system.

I’m not sure if that’s what you’re referring to with a ‘toolchain-based’ build process or not. The reason I think it might be is because I don’t think this would require big changes. We’d just have to install products after swift, then install the target libraries as we go. Foundation could even keep on generating build.ninja in-tree, sadly.

There isn’t much difference between supporting one cross-host or supporting n cross-hosts. I would prefer ’n’, if we could get a nice argument syntax for it, because it’s more convenient to integrate with other tools, but it’s not a big deal. Actually, I have an idea for a better build-script argument syntax which would scale to NxM cross-compiled hosts and targets elegantly, but that’s a topic for another day.

Karl

On 3 Jun 2016, at 22:31, Daniel Dunbar via swift-dev <swift-dev@swift.org> wrote:

On Jun 3, 2016, at 1:14 PM, Saleem Abdulrasool <compnerd@compnerd.org <mailto:compnerd@compnerd.org>> wrote:

On Wed, Jun 1, 2016 at 2:18 PM, Daniel Dunbar via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:
Hi all,

The current build process for the overall Swift project (i.e., the compiler + associated projects like Foundation, XCTest, and SwiftPM) relies on each project having dependencies on the built artifacts of previously built projects. Those dependencies are currently communicated to each project in the chain through an ad hoc set of arguments to the individual project's build process. This has been painful to maintain, and makes it hard to reason about the environment that each of the associated projects are building within.

Instead, I would like to move towards what I have been calling a "toolchain-based" build process.

In this model:

1. The entire build process will be organized as a sequential, incremental construction of a complete toolchain.
        - Each individual project will build and copy its products into a staging area.
2. The build script will always create a composed toolchain.
        - It will start with an empty toolchain, and merge in the content from each project as it builds.
        - This will use rsync & hard-links to avoid needing to stay fast.
3. Each individual project build will just use the composed toolchain to build.
        - This will replace the grab bag of options we use to communicate between the projects.
4. At the end of the build, we will have constructed a complete toolchain which can be installed (or used with Xcode).

Aside from simplifying the overall build process, this has a couple very nice upsides:

1. At the end of the build, the user has a complete toolchain. They can install it, or use it as they would a distributed snapshot. This is very beneficial for people who are only building the Swift project to get a more recent version of the compiler.

2. Each individual project can be built using the "official" build process with a downloaded snapshot (assuming it is of a new enough version). This is very beneficial for easing contribution to projects like Foundation, XCTest, and SwiftPM which are pure Swift and have fast build times, but which currently build in Swift CI using a very different process from the snapshot-based workflow.

Concrete details:

1. I do not plan to change the actual install process in the short term. The actual install process used today relies on the CMake-based install process for some projects (most importantly Swift) and isn't suitable for use in this fashion (where incremental development speed is of high importance). Instead, my plan is to teach the build script itself how to assemble the staging area for each individual project, with the long term goal of using that for the official install process instead of the CMake-based process.

2. My plan is that the build script would only support building one "primary product" (i.e. toolchain). That product may itself be a complete cross compiler toolchain with support for multiple platforms, but the expectation is that users would invoke the build script multiple times if building multiple toolchains. However, to support Canadian Cross [1] build scenarios the build script may need to manage the construction of two products, the primary toolchain and the intermediate (cross compiling) toolchain used to build the artifacts in the primary toolchain.

3. As a concrete example of a problem this solves, SwiftPM currently doesn't test its `swift-test` product in Swift CI, because that product requires using all of the toolchain stack (swift, Foundation, and XCTest), but the product itself is only intended to be used as part of a concrete toolchain. As such, it doesn't know how to operate correctly when given the piecemeal build products for each of those projects, and teaching it to do so would be very cumbersome for us to maintain.

Feedback welcome!

Overall, I think that this would be a great change.

I am however slightly confused on why the build script needs to be concerned about candian cross at all if it can be taught how to use a specified toolchain.

Because these (standard at least as per auto tools) terms can be confusing:

[1] build - the platform where things are being built
[2] host - the platform where the generated binaries will be run
[3] target - the platform where the binaries generated by the compiler on the host platform will run

It should be possible to shift the burden on the user.

Why would we _want_ to shift the burden to the user?

From the user's perspective, they want to end up with a toolchain that compiles `Host -> Target`... I see it as an implementation detail (of the `build-script`) that producing what the user wants requires us to build the intermediate `Build -> Target` toolchain.

  They would be responsible for the first stage of the canadian cross (to build a toolchain capable of generating binaries for the host platform). Then invoke the build script a second time with the toolchain they just built to cross-compile the cross-compiling toolchain.

The tricky part of the canadian cross situation isn't the toolchain capable of generating binaries for the host, it is the toolchain capable of generating binaries for the target, but which needs to run on the build host. This is needed when you need to produce runtime libraries that will execute on the Target, but you are simultaneously trying to construct a cross compiler for a Host which you cannot actually execute. This is when you need the extra `Build -> Target` toolchain, and is the situation I was referring to.

- Daniel

Or did I completely misunderstand your detail point 2 and this is the expectation?

- Daniel

- Daniel

[1] Cross compiler - Wikipedia

_______________________________________________
swift-dev mailing list
swift-dev@swift.org <mailto:swift-dev@swift.org>
https://lists.swift.org/mailman/listinfo/swift-dev

--
Saleem Abdulrasool
compnerd (at) compnerd (dot) org

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

Daniel_Dunbar · 2016-06-11T03:38:41.000Z

Hi Karl,

We can’t separate building for ‘Build’ and building for an arbitrary
‘Host’. For those who don’t know, this is how cross-compiling works with
Swift (this isn’t off-topic, I’m going to bring it back firmly to this
proposal):

Currently, when you install the compiler, the standard library is in
(prefix)/lib/swift/(Target)/(arch). On OSX, we cross-compile the standard
library for the iOS targets, so you’ll see subfolders for ‘macosx’,
‘iphoneos’, etc in there. All you need to cross-compile in Swift is the
standard library; swiftmodules don’t need ‘ar’, so libraries don’t even
need any binutils to cross-compile, which is refreshing. A cross-linker
*is* unfortunately still required for executables (come on, LLD!).

Grab the standard library from a linux machine and try:
echo 'print("hello, world")' | ./swiftc —target=x86-unknown-linux-gnu
-resource-dir {PATH_TO_LINUX_STDLIB} -emit-module -

So when we’re cross-compiling the swift portions of the standard-library,
we’re using the swift compiler we just made for Build, telling it to use
Host’s lib/swift directory, and there it will find the correct subdirectory
for Target. My native compiler for my OSX Build machine knows not to look
for the Linux host’s own Target in swift-macosx-x86_64/lib/swift, but in
swift-linux-armv7/lib/swift.

—

Okay, so that’s out of the way.

There is an actual problem with how we build/configure the target
libraries (Foundation, XCTest and libdispatch). We currently configure them
like every other product (for each host), instead of for each target. A
consequence of this is, for example, that we won’t try to build Foundation
for cross-compile targets such as Android. What we should do is this:

For each host:
-> Build cmark/llvm/swift
-> For each target of this host:
-> -> Build Foundation/XCTest/libdispatch
-> Build swiftpm

When you look at it like this, from a tools // target-libraries
perspective (swiftpm is an exception; it’s a host tool, but it depends on
Foundation), I came up with a different idea (possibly, I’m not sure), that
might also help the dependency issue, although I don’t know the specifics
of it: I would suggest that after building swift, we install it, then build
the target libraries and install them as we do so.

That would give us a stable standard-library (lib/swift) location which we
could treat like an installed system. So when Foundation builds (for each
Target now), it sets -resource-dir to the installed Host copy of swift, and
the Build swift compiler can find Target’s standard library (yeah, I know).
It installs itself in there. Then XCTest comes along, it sets the same
-resource-dir, so now it can find Foundation just with ‘import Foundation’
and nothing else. In other words: every product after the swift compiler
can assume its dependencies are like they’ve been installed on the system.

I’m not sure if that’s what you’re referring to with a ‘toolchain-based’
build process or not. The reason I think it might be is because I don’t
think this would require big changes. We’d just have to install products
after swift, then install the target libraries as we go. Foundation could
even keep on generating build.ninja in-tree, sadly.

Yup, this is exactly what I was referring to (although for different
reasons).

- Daniel

···

On Fri, Jun 10, 2016 at 2:38 AM, Karl via swift-build-dev < swift-build-dev@swift.org> wrote:

There isn’t much difference between supporting one cross-host or
supporting n cross-hosts. I would prefer ’n’, if we could get a nice
argument syntax for it, because it’s more convenient to integrate with
other tools, but it’s not a big deal. Actually, I have an idea for a better
build-script argument syntax which would scale to NxM cross-compiled hosts
and targets elegantly, but that’s a topic for another day.

Karl

On 3 Jun 2016, at 22:31, Daniel Dunbar via swift-dev <swift-dev@swift.org> > wrote:

On Jun 3, 2016, at 1:14 PM, Saleem Abdulrasool <compnerd@compnerd.org> > wrote:

On Wed, Jun 1, 2016 at 2:18 PM, Daniel Dunbar via swift-dev < > swift-dev@swift.org> wrote:

Hi all,

The current build process for the overall Swift project (i.e., the
compiler + associated projects like Foundation, XCTest, and SwiftPM) relies
on each project having dependencies on the built artifacts of previously
built projects. Those dependencies are currently communicated to each
project in the chain through an ad hoc set of arguments to the individual
project's build process. This has been painful to maintain, and makes it
hard to reason about the environment that each of the associated projects
are building within.

Instead, I would like to move towards what I have been calling a
"toolchain-based" build process.

In this model:

1. The entire build process will be organized as a sequential,
incremental construction of a complete toolchain.
        - Each individual project will build and copy its products into
a staging area.
2. The build script will always create a composed toolchain.
        - It will start with an empty toolchain, and merge in the
content from each project as it builds.
        - This will use rsync & hard-links to avoid needing to stay fast.
3. Each individual project build will just use the composed toolchain to
build.
        - This will replace the grab bag of options we use to
communicate between the projects.
4. At the end of the build, we will have constructed a complete toolchain
which can be installed (or used with Xcode).

Aside from simplifying the overall build process, this has a couple very
nice upsides:

1. At the end of the build, the user has a complete toolchain. They can
install it, or use it as they would a distributed snapshot. This is very
beneficial for people who are only building the Swift project to get a more
recent version of the compiler.

2. Each individual project can be built using the "official" build
process with a downloaded snapshot (assuming it is of a new enough
version). This is very beneficial for easing contribution to projects like
Foundation, XCTest, and SwiftPM which are pure Swift and have fast build
times, but which currently build in Swift CI using a very different process
from the snapshot-based workflow.

Concrete details:

1. I do not plan to change the actual install process in the short term.
The actual install process used today relies on the CMake-based install
process for some projects (most importantly Swift) and isn't suitable for
use in this fashion (where incremental development speed is of high
importance). Instead, my plan is to teach the build script itself how to
assemble the staging area for each individual project, with the long term
goal of using that for the official install process instead of the
CMake-based process.

2. My plan is that the build script would only support building one
"primary product" (i.e. toolchain). That product may itself be a complete
cross compiler toolchain with support for multiple platforms, but the
expectation is that users would invoke the build script multiple times if
building multiple toolchains. However, to support Canadian Cross [1] build
scenarios the build script may need to manage the construction of two
products, the primary toolchain and the intermediate (cross compiling)
toolchain used to build the artifacts in the primary toolchain.

3. As a concrete example of a problem this solves, SwiftPM currently
doesn't test its `swift-test` product in Swift CI, because that product
requires using all of the toolchain stack (swift, Foundation, and XCTest),
but the product itself is only intended to be used as part of a concrete
toolchain. As such, it doesn't know how to operate correctly when given the
piecemeal build products for each of those projects, and teaching it to do
so would be very cumbersome for us to maintain.

Feedback welcome!

Overall, I think that this would be a great change.

I am however slightly confused on why the build script needs to be
concerned about candian cross at all if it can be taught how to use a
specified toolchain.

Because these (standard at least as per auto tools) terms can be confusing:

[1] build - the platform where things are being built
[2] host - the platform where the generated binaries will be run
[3] target - the platform where the binaries generated by the compiler on
the host platform will run

It should be possible to shift the burden on the user.

Why would we _want_ to shift the burden to the user?

From the user's perspective, they want to end up with a toolchain that
compiles `Host -> Target`... I see it as an implementation detail (of the
`build-script`) that producing what the user wants requires us to build the
intermediate `Build -> Target` toolchain.

  They would be responsible for the first stage of the canadian cross (to
build a toolchain capable of generating binaries for the host platform).
Then invoke the build script a second time with the toolchain they just
built to cross-compile the cross-compiling toolchain.

The tricky part of the canadian cross situation isn't the toolchain
capable of generating binaries for the host, it is the toolchain capable of
generating binaries for the target, but which needs to run on the build
host. This is needed when you need to produce runtime libraries that will
execute on the Target, but you are simultaneously trying to construct a
cross compiler for a Host which you cannot actually execute. This is when
you need the extra `Build -> Target` toolchain, and is the situation I was
referring to.

- Daniel

Or did I completely misunderstand your detail point 2 and this is the
expectation?

- Daniel

- Daniel

[1] Cross compiler - Wikipedia

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

--
Saleem Abdulrasool
compnerd (at) compnerd (dot) org

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

_______________________________________________
swift-build-dev mailing list
swift-build-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-build-dev