SE-0387: Cross-Compilation Destination Bundles

Max_Desiatov · 2023-02-10T09:32:34.802Z

Ok, that's interesting, I'd expect it to switch to a peer /usr/lib/swift_static directory automatically based on whether --no-static-swift-stdlib or --static-swift-stdlib are passed even when /usr/lib/swift is a -resource-dir, isn't this the current behavior? Do we really need to make the path for static resources explicit?

Update:

kateinoigakukun:

Currently, swiftResourcesPath takes a path to the resource path in the bundle, but the resource path is not only one thing. In non-CC context, swift driver determines one of the resource paths lib/swift and lib/swift_static based on given options (-static-stdlib, -static-executable, ..) by assuming they are placed in relative to the compiler executable.

However, when we pass the resource path to the compiler by -resource-dir, lib/swift or lib/swift_static should be determined by those who are passing the option. (in this case, SwiftPM has the responsibility)

I see your point now, I need to have a closer look at how SwiftPM handles this to give you my opinion on this. But at a second glance this makes sense to me overall.

Update 2:

kateinoigakukun:

When destination.json has the following entry:
      "swiftResourcesPath": {
        "static": "./usr/lib/swift_static",
        "dynamic": "./usr/lib/swift"
      }
Then:

swift build --destination XXX --no-static-swift-stdlib should pass -resource-dir ./usr/lib/swift to the driver

swift build --destination XXX --static-swift-stdlib should pass -resource-dir ./usr/lib/swift_static to the driver

I've introduced "swiftStaticResourcesPath" in Update SE-0387: Cross-Compilation Destination Bundles by MaxDesiatov · Pull Request #1942 · apple/swift-evolution · GitHub to handle this, converting it to a separate dictionary seems like an overkill. Some platforms may support only static linking, some only dynamic, having separate properties without a new dictionary seems more suitable to me.

Thus currently proposed destination.json schema is:


```json5
{
  "schemaVersion": "3.0",
  "runTimeTriples": [
    "<triple1>": {
      "sdkRootPath": "<a required path relative to `destination.json` containing SDK root>",
      // all of the properties listed below are optional:
      "swiftResourcesPath": "<a path relative to `destination.json` containing Swift resources for dynamic linking>",
      "swiftStaticResourcesPath": "<a path relative to `destination.json` containing Swift resources for static linking>",
      "includeSearchPaths": ["<array of paths relative to `destination.json` containing headers>"],
      "librarySearchPaths": ["<array of paths relative to `destination.json` containing libraries>"],
      "toolsetPaths": ["<array of paths relative to `destination.json` containing toolset files>"]
    },
    // a destination can support more than one run-time triple:
    "<triple2>": {
      "sdkRootPath": "<a required path relative to `destination.json` containing SDK root>",
      // all of the properties listed below are optional:
      "swiftResourcesPath": "<a path relative to `destination.json` containing Swift resources for dynamic linking>",
      "swiftStaticResourcesPath": "<a path relative to `destination.json` containing Swift resources for static linking>",
      "includeSearchPaths": ["<array of paths relative to `destination.json` containing headers>"],
      "librarySearchPaths": ["<array of paths relative to `destination.json` containing libraries>"],
      "toolsetPaths": ["<array of paths relative to `destination.json` containing toolset files>"]
    }
    // more triples can be supported by a single destination if needed, primarily for sharing files between them.
  ]
}

Max_Desiatov · 2023-02-10T10:07:35.449Z

kateinoigakukun:

swift driver determines one of the resource paths lib/swift and lib/swift_static based on given options (-static-stdlib, -static-executable, ..) by assuming they are placed in relative to the compiler executable

nit: this seems to be located within frontend logic, not driver logic, but seems that it is relative to the compiler executable, not to a given -resource-dir value.

Max_Desiatov · 2023-02-10T15:15:04.658Z

Finagolfin:

OK, so good for cases where you want to reuse certain tool configs without any flags. To really enable that, you may want to add a way to use just the tools from a toolset, but wipe or overwrite all the command-line flags that come with that toolset.

If there's a need for wiping or overwriting all CLI options specified by a toolset, just copy it and redefine its extraCLIOptions values accordingly. Thus even when you have both toolsets specified in a single invocation, if the new comes after the original one it will overwrite options of the original.

Finagolfin:

I notice you skipped over my concern about the system PATH lookups.

I've clarified my stance on this in one of the previous posts and I don't have anything to add. I'm currently not convinced that this should be codified in the proposal, especially as it's not an already established behavior and not something I've seen to be frequently requested. I'm open to reviewing concrete examples where the existing behavior caused active harm to reconsider.

Finagolfin:

Yes, but in your example destination after the update to this proposal, jammy supports multiple runtime triples, so this command of simply specifying a destination alone will fail.

I like all these CLI shortcuts you provide, I'm just saying lay out the full command needed when specifying the destination alone won't suffice.

Good catch, I've updated to swift build --destination ubuntu_focal, which in the context of the example provides only one run-time triple.

kabiroberai:

Will there be a way to specify an entire bin directory like destination.json's toolchain-bin-dir? If not, I agree with @Finagolfin in that we need more options to configure individual tools like ar.

Check out the updated proposal text, it does mention the new librarian toolset field for ar, and if the proposal is accepted with the toolsets feature, we'll be adding new properties for other tools as needed.

kabiroberai:

Or does this give toolchain authors the ability to define custom run-time triples? And if we can define custom triples, is there a way we can make them "inherit" from existing ones?

It does not. Supporting an arbitrary triple is not just a matter of adding cases to an enum. All of the triple components (CPU architecture, OS, object format, libc flavor etc) need to be supported in all of the different parts of the toolchain and core libraries and each step requires a non-trivial amount of work and integration testing, as it does for any language. I'm personally not sure if this can be specified in a Swift Evolution proposal, it's primarily an implementation detail of the toolchain and core libraries.

kabiroberai:

For example, SPM's Triple doesn't know about arm64-apple-iphoneos (even though swift-driver does) so it'd be ideal if I could define said triple to inherit from arm64-apple-darwin.

Sorry, I'm not sure what "inherit" is supposed to mean in this context. If you think this is something that should be fixed in SwiftPM specifically, please create an issue or a PR on GitHub and we can discuss it there.

Finagolfin · 2023-02-12T06:58:30.207Z

If there's a need for wiping or overwriting all CLI options specified by a toolset, just copy it and redefine its extraCLIOptions values accordingly.

That invalidates the claim of toolset reuse in the proposal, which I was skeptical of to begin with. The problem with most toolsets is that they will have flags specific to a particular usecase, so they will not be reusable, unless you can easily wipe their CLI flags as I suggest. If you have to write a whole new toolset config to add new flags instead, as you now suggest, there is no reusability, as you might as well write the config from scratch at that point.

Anyway, this is just a suggestion on the toolsets expansion of the bundle proposal, not too important.

I've clarified my stance on this in one of the previous posts

That would be where you wrote, "This seems wrong to me," not that confident.

I'm currently not convinced that this should be codified in the proposal, especially as it's not an already established behavior and not something I've seen to be frequently requested.

How many people do you know using the previous destination JSON config? I suspect it numbers on one hand, and I'm the one using it the most.

The point of this bundles proposal is to expand that greatly, and disabling these system lookups would keep them safe rather than sorry.

Good catch, I've updated to swift build --destination ubuntu_focal , which in the context of the example provides only one run-time triple.

No, it simply does not specify how many runtime triples it has. I just don't think it's good to write a proposal only listing CLI shortcuts, but not a single example of the full command.

Max_Desiatov · 2023-02-12T14:56:04.971Z

Finagolfin:

I just don't think it's good to write a proposal only listing CLI shortcuts, but not a single example of the full command.

These are not CLI shortcuts, but actual examples of full commands that I run locally to test the prototype implementation.

xwu · 2023-02-12T17:51:13.726Z

Love to see how this proposal has evolved since the pitch phase.

I see the point about moving to “build-time triple” and “run-time triple” terminology, but there’s still a lot of places that use “host” and “destination” in this text: is there a way to explain the distinction?

Some nits with respect to the schema:

What do you think of headerSearchPaths to parallel librarySearchPaths—the proposed spelling includeSearchPaths can be misread as a imperative ("include these search paths for...anything")
In general, when a noun is used to modify another noun in English, one prefers the singular (unless there is some unavoidable ambiguity in meaning). For example, as you do here, "library search paths" not "libraries search paths." For the same reason, in standard English one would prefer "Swift resource path" over "Swift resources path." Is there some ambiguity here that caused you to choose swiftResourcesPath with the plural specifically?

xwu · 2023-02-12T18:01:02.062Z

Max_Desiatov:

I've clarified my stance on this in one of the previous posts and I don't have anything to add. I'm currently not convinced that this should be codified in the proposal, especially as it's not an already established behavior and not something I've seen to be frequently requested. I'm open to reviewing concrete examples where the existing behavior caused active harm to reconsider.

Following up in a subsequent post because the forums software for some reason has disabled my return key, for which reason the rest of this feedback will be in one big paragraph (sorry). I do think @Finagolfin brings up a valid point of concern here that, with better cross-destination compilation support, silently falling back to system tools could lead to difficult-to-diagnose unexpected results. The introduction of a new cross-compilation schema would be an ideal time to explore new restrictions that, if found to be conservative, could always be relaxed later. As a middle way, at least, SPM could warn loudly when it chooses to use a fallback tool when performing cross-compilation, with perhaps an optional toggle in the schema to disallow it altogether.

Max_Desiatov · 2023-02-12T20:33:57.455Z

xwu:

SPM could warn loudly when it chooses to use a fallback tool when performing cross-compilation

I'm willing to take a bet that in practice 90%+ of destination bundles won't be completely self-contained and will need to fall back on tools from PATH. There's little value in not reusing tools like git, zip etc that are already installed on the user's machine and work perfectly fine. In fact, all of the destinations I've produced while implementing this proposal worked that way.

The main benefit is that destination bundles can be smaller. For example, when cross-compiling from macOS to Linux, that only tool you need that's not included with the Xcode toolchain is a linker that supports ELF object format. Thus the vast majority of bundles for cross-compiling in that direction will need a toolset with a single entry and include a single binary for the Swift toolchain to work as needed.

If tools from the existing environment can't be picked automatically, it will require destination bundles to become much larger by including all of the tools, and that implies hundreds of megabytes of binaries (if not more in a typical scenario) copied around, duplicating those available in PATH. As an alternative, users of destinations will need to maintain their own explicit toolset files that enumerate every single binary that a toolchain might use. And there's no way to enforce that none of the tools would launch something from PATH anyway.

xwu:

The introduction of a new cross-compilation schema would be an ideal time to explore new restrictions that, if found to be conservative, could always be relaxed later

The issue is that we already found it too conservative while trying to implement it. This restriction is not cleanly enforceable, while proving to be burdensome from maintenance and DX perspective.

filip-sakel · 2023-02-12T21:22:48.681Z

Max_Desiatov:

practice 90%+ of destination bundles won't be completely self-contained and will need to fall back on tools from PATH. There's little value in not reusing tools like git, zip etc that are already installed on the user's machine and work perfectly fine. In fact, all of the destinations I've produced while implementing this proposal worked that way.

What other system tools does SwiftPM use?

Finagolfin · 2023-02-13T15:19:08.045Z

These are not CLI shortcuts, but actual examples of full commands

It is a shortcut because in the most general case of multiple destinations that refer to the same triple, none of the commands you've listed so far will work.

I'm willing to take a bet that in practice 90%+ of destination bundles won't be completely self-contained and will need to fall back on tools from PATH .

If you limit it to tools that generate code, it's more like the opposite.

There's little value in not reusing tools like git , zip etc that are already installed on the user's machine and work perfectly fine.

Since nobody has suggested not using those particular non-code-generating tools, and I specifically said it would be fine to keep using those, it is disingenuous of you to keep mentioning them.

If tools from the existing environment can't be picked automatically, it will require destination bundles to become much larger by including all of the tools, and that implies hundreds of megabytes of binaries (if not more in a typical scenario) copied around, duplicating those available in PATH .

This is a giant exaggeration, it won't be more than tens.

As an alternative, users of destinations will need to maintain their own explicit toolset files that enumerate every single binary that a toolchain might use.

No, most tools are already packaged in the Swift toolchain and so will not need to be enumerated in a toolset.

And there's no way to enforce that none of the tools would launch something from PATH anyway.

Sure, but that's not an excuse for making the problem worse by not cleaning up what we can.

The issue is that we already found it too conservative while trying to implement it.

I find that hard to believe: what tool did you have a problem with?

This restriction is not cleanly enforceable,

I've pointed out exactly where in the SPM source many tools are looked up: it would be trivial to disable those, while I fully grant there are others we don't know about and cannot easily control.

while proving to be burdensome from maintenance and DX perspective.

I don't think it will be a burden and am prepared to chip in myself on getting SPM working this way.

Let me emphasize that most bundles will mostly use the existing tools from the Swift toolchain or supply their own code-generating tools, ie not git, zip, etc but the assembler or linker. This whole discussion is about what to do about the few exceptions, where for some reason SPM can't find some tool and currently falls back to the system PATH.

I believe it's a good precaution to disable such system lookups that we know about in those few exceptional situations, but maybe you disagree. I fully grant that it would not be a promise of no system lookups, as I originally noted, "It is not clear what all the tools pulled in are: I doubt even the toolchain authors know at this point."

soumyamahunt · 2023-04-10T14:18:56.922Z

I am excited to see the efforts for making multi-platform development easier. However I have some questions:

Will in the future, the destination bundles be able to provide additional package product definitions, i.e. iOSApplication product for ios destination, and similar for android apps?

Currently building iOS apps with playground creates a swift package that exposes iOSApplication product defined in AppleProductTypes. Having a destination bundle level implementation might reduce the setup required for cross-platform development.
If compiler directive os() is used in Package.swift, will it check for the destination platform or host platform or solely dependent upon the implementation of destination bundles?

Max_Desiatov · 2023-04-11T11:22:06.534Z

soumyamahunt:

Will in the future, the destination bundles be able to provide additional package product definitions, i.e. iOSApplication product for ios destination, and similar for android apps?

Currently building iOS apps with playground creates a swift package that exposes iOSApplication product defined in AppleProductTypes. Having a destination bundle level implementation might reduce the setup required for cross-platform development.

Xcode, Playgrounds, and iOS SDK are out of scope of Swift Evolution. You can file an FB feature request if you think there's something that can be improved on that front.

soumyamahunt:

If compiler directive os() is used in Package.swift, will it check for the destination platform or host platform or solely dependent upon the implementation of destination bundles?

This behavior is already established and we're not changing it here. Using current proposal's nomenclature, this will check for the build-time OS. If you need run-time platform checks, use conditional target dependencies as described in SE-0273.

wowbagger · 2023-04-12T20:19:02.678Z

+1 in general, but I echo @Finagolfin's concern against silent fallback to system PATH. Also I prefer headerSearchPaths as @xwu suggested.

I don't have much else to add to this review, except for an out-of-scope-ish question, which is somewhat relevant to but different from the "SwiftPM Plugins for Remote Running, Testing, Deployment, and Debugging" future direction: Could this feature serve as a possible/suitable foundation for implementing distributed compilation and/or automatic offloading of Swift programs?

Max_Desiatov · 2023-04-12T20:27:04.623Z

wowbagger:

concern against silent fallback to system PATH

As I mentioned above, this is not a silent fallback. We're following the expected system behavior here. From any perspective it's surprising that PATH would not be doing what it's documented and supposed to do. If someone wants that variable not to have a certain effect, they should reassign it to their desired value. An empty string I guess, with PATH="" if that's needed?

wowbagger:

Could this feature serve as a possible/suitable foundation for implementing distributed compilation and/or automatic offloading of Swift programs?

I'm not sure it could, this seems to be orthogonal to cross-compilation. Maybe one could come up with a wrapper similar to sccache and create a toolset that utilizes it?

wowbagger · 2023-04-13T19:31:17.160Z

Max_Desiatov:

As I mentioned above, this is not a silent fallback. We're following the expected system behavior here. From any perspective it's surprising that PATH would not be doing what it's documented and supposed to do. If someone wants that variable not to have a certain effect, they should reassign it to their desired value. An empty string I guess, with PATH="" if that's needed?

By my understanding, the concern is that system tools are out of the package author's control, and any difference in behaviour across different systems' tools might break in unexpected and difficult-to-diagnose ways. Also what happens if the system doesn't have tools such as Git installed? It's not very likely, but certainly possible.

Max_Desiatov:

I'm not sure it could, this seems to be orthogonal to cross-compilation. Maybe one could come up with a wrapper similar to sccache and create a toolset that utilizes it?

I should've clarified that when I said distributed compilation, I was thinking about that across a heterogeneous system, in which each machine cross-compiles parts of the whole thing.

Max_Desiatov · 2023-04-18T11:39:27.640Z

wowbagger:

By my understanding, the concern is that system tools are out of the package author's control, and any difference in behaviour across different systems' tools might break in unexpected and difficult-to-diagnose ways.

Yes, but this was always the case and there's no good way to enforce it in SwiftPM itself right now, and AFAIU in the foreseeable future. Even if hypothetically one prevents SwiftPM from using PATH when using a cross-compilation bundle, we don't have enough control over the rest of the tools, like clang etc, those can look into PATH whenever they want. What's the point in requiring something to be enforced if there are even no mechanisms to enforce it consistently?

wowbagger:

Also what happens if the system doesn't have tools such as Git installed? It's not very likely, but certainly possible.

Not sure how that's relevant to cross-compilation? Git is an explicit requirement of SwiftPM. A installation of Swift with SwiftPM but without Git should be considered broken, but I don't think this should be a concern on the level cross-compilation tools. It's a responsibility of the user installing the "top-level" toolchain to provide all of the required dependencies.

Finagolfin · 2023-04-19T08:51:06.285Z

What's the point in requiring something to be enforced if there are even no mechanisms to enforce it consistently?

We cannot easily "enforce" it for all possible invocations of tools from the system PATH by the toolchain, but we can easily make sure that SPM does not look in the PATH for tools like the linker when cross-compiling, thus lowering but not eliminating the chance of system tool lookups.

It's not all or nothing, I'm simply advocating picking that low-hanging fruit.

Max_Desiatov · 2023-04-19T09:20:18.721Z

I understand that, but my argument is for consistency and predictability that "all or nothing" approach gives us.

PATH has an established behavior that users come to expect. If it isn't applied consistently then we fall into a rabbit hole of keeping track of all the places it's applied or not applied, surfacing these differences in behavior in our documentation, and getting it displayed to users properly in diagnostics if something goes wrong. I don't think this is worth the maintenance burden, especially if users alternatively can set PATH to an empty string on their side and get vastly more predictable behavior when needed.

mishal_shah · 2023-04-28T04:11:54.079Z

[Accepted in principle] SE-0387: Cross-Compilation Destination Bundles Proposal Reviews

The review of SE-0387: Cross-Compilation Destination Bundles ended on February 14th, 2023, and the core team has decided to accept the proposal in principle the following modifications. Also, return it for re-review focused on naming convention. There is a new review thread for that discussion. In toolset files the name of the extraFlags property that provides additional command-line options for each tool seemed misleading. Its value may contain not only flags, but any command-line options. ex…

mishal_shah · 2023-04-28T04:12:49.789Z