Confused by unsafe flags being disallowed in dependencies

[tgoyne (Thomas Goyne)]

That notably isn't an unstable Swift flag; it's a perfectly stable clang flag. Not being able to set clang and linker flags has been the main pain-point for us, incidentally.

---

Does the Swift frontend consider 100% of it's command-line interface outside of the handful of things exposed by SPM to be unstable and experimental?

[NeoNacho (Boris Buegling)]

@tgoyne

Does the Swift frontend consider 100% of it's command-line interface outside of the handful of things exposed by SPM to be unstable and experimental?

No, I believe we could expose more driver flags, only frontend flags are a concern in terms of compatibility. We probably want to be careful with flags that could lead to breaking the build or dependency model (e.g. allowing arbitrary search paths), but there are still a lot of things on the table to potentially be added.

In fact, the original proposal promised adding more settings, we just haven't actually done that. One we discussed internally last year was e.g. -enable-bare-slash-regex which currently can't be used in a package.

If it helps to add first-class API for some missing settings that are commonly used and we can come up with a list on this thread, I am definitely open to writing up a proposal and implementation to add them. I have seen the occasional specific requests here and there, but nothing had risen to the point that it became a priority so far. That may have been the wrong approach though and we should just start adding what's needed.

[tgoyne (Thomas Goyne)]

I think ideally it should be up to the driver (or the various tools invoked by the driver) to determine what flags are safe. There could be something like a -package-safe-flags-only flag which tells the driver that all subsequent flags are from packages rather than SPM itself, and it would reject any not flagged as PackageSafe in Options.td. This would avoid the need for SPM to be able to parse the command-line flags and means that there's always a single source of truth for what compiler flags are available rather than duplicating that list.

[xwu (Xiaodi Wu)]

@NeoNacho

In fact, the original proposal promised adding more settings, we just haven't actually done that. One we discussed internally last year was e.g. -enable-bare-slash-regex which currently can't be used in a package.

This should be possible with Swift 5.8 using the spelling .upcomingFeatures(["BareSlashRegexLiterals"]), should it not?

[hassila (Joakim Hassila)]

@NeoNacho

One thing that I am wondering about is whether it would be useful to allow unsafe flags for 0.x and prerelease versions since those carry inherent instability with them which makes any compatibility concerns way less severe. Would that be helpful for at least some of the use cases in this thread?

It'd help somewhat, but not completely (see e.g. the swift-atomics use case, there could be other similar things like the compiler bug I pointed out as well where you definitely want to be able to work around it to support a new/old toolchain even after you've shipped 1.0). I think it might just push people to never do 1.0 in that case, which would be a bit sad.

@NeoNacho

Another idea that was recently discussed was offering additional, inherently unstable APIs in the PackageDescription via @_spi. I wonder if there's some potential there to offer an API that can be opted in by versioned packages in a way that makes it very clear to authors that they are treading on somewhat unsupported grounds.

I think that could go a long way to alleviate the issues, as some escape hatch for when the existing functionality of SwiftPM just not enough would be useful - just the word unsafe in unsafeFlags at least tells me that I need to be careful and consider the long term effects of using them, but if even more clarity is desired, it's fine by me - in this case I just want us to be able to be unblocked and use SwiftPM dependencies properly even if we need to use some unsafe flag.

@NeoNacho

If it helps to add first-class API for some missing settings that are commonly used and we can come up with a list on this thread, I am definitely open to writing up a proposal and implementation to add them. I have seen the occasional specific requests here and there, but nothing had risen to the point that it became a priority so far. That may have been the wrong approach though and we should just start adding what's needed.

That would be super nice of course, thanks a lot for offering to consider that.

I just did a quick canvassing of the types of unsafe flags we've used recently (some might be much more useful implemented as different SwiftPM first-class features for sure, but just to give some input and to provide some background context, in no order of priority:

1. The ability to enable library evolution mode and module versioning, related to
   Dynamic library support on Linux with library evolution · Issue #5714 · apple/swift-package-manager · GitHub
   and
   Availability annotations for third party libraries when using Library evolution/ resilience · Issue #60458 · apple/swift · GitHub

            swiftSettings: [.unsafeFlags(["-enable-library-evolution",
                                         "-emit-module-interface",
                                          "-user-module-version", "1.0"

2. Additional linker settings

  linkerSettings: [
                // for M1 based Macs
                .unsafeFlags(["-Xlinker", "-L/opt/homebrew/lib"], .when(platforms: [.macOS])),
                // for Intel based Macs
                .unsafeFlags(["-Xlinker", "-L/usr/local/lib"], .when(platforms: [.macOS])),
            ]

3. Overriding optimization level by target
   This is very commonly desired. Several times we want to do this, e.g. for dependencies of a Command plugin so that the executable dependency will be built optimised always when used by consumers of the package (I view the command plugin and related dependencies as part of the toolchain, just as I want my swift compiler to have been built with optimization turned on, I want the helper tool for the command plugin that processes tons of json to be optimised for a 2-3x speedup of post processing) - we have run into this in various settings, not just in this case (sometimes a specific part of the code will make debug runs so slow to be unusable, so it makes sense to turn that target to optimized, and conversely, something might need debug stuff).

SwiftPM always rebuilds command plugins in release configuration

So, short background: I have a command plugin that has a tool as a dependency that performs the heavy work. I extract the tool from the command line plugin using: let benchmarkTool = try context.tool(named: "BenchmarkTool") print("\(benchmarkTool.path)") This gives the path: .../package-benchmark/.build/arm64-apple-macosx/debug/BenchmarkTool Now, this is getting the tool with the same build configuration that the command plugin is built with (debug), which was running prett…

swiftSettings: [SwiftSetting.unsafeFlags(["-Osize"])]

4. Enabling experimental features
   (as pointed out by @xwu this now will be in 5.8 thanks to yourself, @Douglas_Gregor and others)

                .unsafeFlags(["-Xfrontend", "-enable-experimental-distributed"]),
            ]

5. Working around compiler issues

swiftSettings: [
   .unsafeFlags(["-Xfrontend", "-validate-tbd-against-ir=none"]), // due to https://bugs.swift.org/browse/SR-15629
 ]

[xwu (Xiaodi Wu)]

@hassila

4. Enabling experimental features
   (as pointed out by @xwu this now will be in 5.8 thanks to yourself, @Douglas_Gregor and others)

Upcoming features are going to be enabled; those are not experimental. Experimental features aren’t intended to be part of that.

[hassila (Joakim Hassila)]

Thanks for the clarification @xwu, I guess then it could be kept as a point of something that is needed and not solved by the upcoming-feature support, as sometime you want to enable an experimental feature even in a package that is versioned after 1.0 (like in our case, distributed actors).

[Torust (Thomas Roughton)]

To add a sixth point to @hassila's list above (a lot of which I've also run into): enabling CPU features can currently only be done with unsafe flags ([SR-11660] Umbrella: function multiversioning and dispatch on CPU features · Issue #54069 · apple/swift · GitHub). This means it's currently not possible to package Swift code that uses e.g. AVX support or hardware FMA.

[hassila (Joakim Hassila)]

Actually, there are two different flags added as it seems:

[SE-0362] Add `enableUpcomingFeature` and `enableExperimentalFeature` Swift settings

So both should (hopefully) be supported with unsafe flags