Confused by unsafe flags being disallowed in dependencies

Thank you! Fixed this with the proposed workaround (.package(path:)). In case anyone is interested, you may have to reexport with @_exported import

1 Like

Was this ever addressed?

The issue is still happening for me.

I've filed a bug about this: [SR-15398] Xcode blocks local-branch SPM packages with unsafe flags in an app targets · Issue #4379 · apple/swift-package-manager · GitHub

Couldn't reply in jira so I'll copy my comment here

I'd also like to add that it could be nice to have an -allow-unsafe-flags flag for xcodeproj target and allow unsafe flags for local packages by default, for example, if I need -Xfrontend -warn-long-expression-type-checking for optimization debugging purposes I'd like to make it work out of the box, without need to wrap my local package into another local package... I'm not even sure it's actually unsafe :new_moon_with_face:

Project structure without a wrapper package is semantically correct and simple

- App.xcworkspace
- App.xcodeproj
- AppPackage

And the wrapper package seems pretty much redundant (actually it does nothing but exposing AppPackage.AppTarget to the App.xcodeproj via _AppPackage._AppTarget (we cant even keep naming for the target))

- App.xcworkspace
- App.xcodeproj
- _AppPackage
- AppPackage

My use case is for linking with private frameworks elegantly by exporting headers via a C target and linking against the relevant framework by passing an unsafe linker flag. By disallowing versioned packages from using unsafe flags, wrappers like this are barred from the semver world and the dependency resolution that comes with it. I understand that a version bump could introduce malicious flags, but I control all ends of the pipeline and would appreciate an option to override this safeguard.

In the meantime I'll be adding the package as a branch-item dependency, like a madman, because there is no alternative that offers the same ease of use for depending packages. GitHub - EricRabil/Paris: SPM-native private framework bindings

Update 22h later: I did not test out adding Paris as a revision-item, which allows me to specify a commit hash. I am now able to depend on specific snapshots of Paris across projects, without worrying about breaking changes.

I still think that being able to use semantic versioning with a package that uses unsafe flags should not be arbitrarily gatekeeped by the package manager. Until SPM is able to cover more advanced targets, there should be a mechanism to at least whitelist specific unsafe flags. Disallowing this outright makes it difficult to push the boundaries of what you can do with Swift.

2 Likes

Just to bump this thread, as running into this issue when lower level versioned libraries are using unsafe flags, it prevents these libs from being used by other packages.

Another vote for having ability to have this check to be a just a warning rather than full bail out.

2 Likes

Just a note here, it is very unlikely that we would ever change the behavior for unsafe flags since allowing arbitrary flags breaks the intended hermetic build model of packages and would render tools-versions and semantic versioning more or less meaningless.

1 Like

i have also run into this countless times and it has led me to conclude that SPM semantic versioning is more or less useless. many packages need "unsafe" configurations, including "official" packages like swift-atomics.

also, let us not pretend that semantic versioning is adhered to rigorously across the ecosystem in the first place.

4 Likes

I think it would be fairly reasonable to allow for opt-in for a given dependency on unsafe flags (not a blanket free-for-all).

It's definitely not arbitrary flags and we'd definitely bump the semantic version of the package if we ever would change those flags.

The longer term nicer option would be if SPM could support "all" needed use cases, but currently it instead becomes a blocker.

One simple example would be this one, where unsafe flags were required to workaround a compiler bug on Linux:

swiftSettings: [
   .unsafeFlags(["-Xfrontend", "-validate-tbd-against-ir=none"]), // due to https://bugs.swift.org/browse/SR-15629
 ]

This was the compiler bug

We have other similar examples related to linkage paths for system installed libraries (perhaps were just holding it wrong, WDIK).

It'd be fantastic to trust us, the users, to opt-in in a responsible way for such cases, as the alternative just now is that we're stuck (now that specific example happily has since been fixed and we could remove the flag, but for a long time that was not an option). A warning is fine even for opt-in, as we do want to clean it up as SPM matures and gains additional capabilities over time, we just don't want to get stuck in the meantime...

3 Likes

I don't actually see any unsafe flags in swift-atomics, am I looking in the wrong place?

Getting a 404 for this.

I don't have any issue with trusting users, my concern is entirely based on whether we can actually guarantee that packages stay working over a longer period of time and that they are portable as much as possible. These are important goals for SwiftPM.

An opt-in makes it somewhat trivial to depend on inherently unstable features like Swift frontend flags which could put us into a situation where the only choices are breaking a chunk of the package ecosystem or having to support features that were supposed to be unsupported in perpetuity.

Unsafe flags could also be non-portable in non-obvious ways, e.g. this just came up recently.

One thing that I am wondering about is whether it would be useful to allow unsafe flags for 0.x and prerelease versions since those carry inherent instability with them which makes any compatibility concerns way less severe. Would that be helpful for at least some of the use cases in this thread?

Another idea that was recently discussed was offering additional, inherently unstable APIs in the PackageDescription via @_spi. I wonder if there's some potential there to offer an API that can be opted in by versioned packages in a way that makes it very clear to authors that they are treading on somewhat unsupported grounds.

Oh, sorry - thought that was public, wanted to show concrete example, but here’s the relevant snippet:

swiftSettings: [
   .unsafeFlags(["-Xfrontend", "-validate-tbd-against-ir=none"]), // due to https://bugs.swift.org/browse/SR-15629
 ]

This was the compiler bug

I’ll need to read and digest the rest of your response and will reply separately - thanks.

on linux, the package needs

 -Xcc -mcx16 -Xswiftc -DENABLE_DOUBLEWIDE_ATOMICS 

to enable double-wide support, which is required for things like AtomicReference.

3 Likes

That notably isn't an unstable Swift flag; it's a perfectly stable clang flag. Not being able to set clang and linker flags has been the main pain-point for us, incidentally.


Does the Swift frontend consider 100% of it's command-line interface outside of the handful of things exposed by SPM to be unstable and experimental?

2 Likes

No, I believe we could expose more driver flags, only frontend flags are a concern in terms of compatibility. We probably want to be careful with flags that could lead to breaking the build or dependency model (e.g. allowing arbitrary search paths), but there are still a lot of things on the table to potentially be added.

In fact, the original proposal promised adding more settings, we just haven't actually done that. One we discussed internally last year was e.g. -enable-bare-slash-regex which currently can't be used in a package.

If it helps to add first-class API for some missing settings that are commonly used and we can come up with a list on this thread, I am definitely open to writing up a proposal and implementation to add them. I have seen the occasional specific requests here and there, but nothing had risen to the point that it became a priority so far. That may have been the wrong approach though and we should just start adding what's needed.

1 Like

I think ideally it should be up to the driver (or the various tools invoked by the driver) to determine what flags are safe. There could be something like a -package-safe-flags-only flag which tells the driver that all subsequent flags are from packages rather than SPM itself, and it would reject any not flagged as PackageSafe in Options.td. This would avoid the need for SPM to be able to parse the command-line flags and means that there's always a single source of truth for what compiler flags are available rather than duplicating that list.

This should be possible with Swift 5.8 using the spelling .upcomingFeatures(["BareSlashRegexLiterals"]), should it not?

2 Likes

It'd help somewhat, but not completely (see e.g. the swift-atomics use case, there could be other similar things like the compiler bug I pointed out as well where you definitely want to be able to work around it to support a new/old toolchain even after you've shipped 1.0). I think it might just push people to never do 1.0 in that case, which would be a bit sad.

I think that could go a long way to alleviate the issues, as some escape hatch for when the existing functionality of SwiftPM just not enough would be useful - just the word unsafe in unsafeFlags at least tells me that I need to be careful and consider the long term effects of using them, but if even more clarity is desired, it's fine by me - in this case I just want us to be able to be unblocked and use SwiftPM dependencies properly even if we need to use some unsafe flag.

That would be super nice of course, thanks a lot for offering to consider that.

I just did a quick canvassing of the types of unsafe flags we've used recently (some might be much more useful implemented as different SwiftPM first-class features for sure, but just to give some input and to provide some background context, in no order of priority:

  1. The ability to enable library evolution mode and module versioning, related to
    Dynamic library support on Linux with library evolution · Issue #5714 · apple/swift-package-manager · GitHub
    and
    Availability annotations for third party libraries when using Library evolution/ resilience · Issue #60458 · apple/swift · GitHub
            swiftSettings: [.unsafeFlags(["-enable-library-evolution",
                                         "-emit-module-interface",
                                          "-user-module-version", "1.0"
  1. Additional linker settings
  linkerSettings: [
                // for M1 based Macs
                .unsafeFlags(["-Xlinker", "-L/opt/homebrew/lib"], .when(platforms: [.macOS])),
                // for Intel based Macs
                .unsafeFlags(["-Xlinker", "-L/usr/local/lib"], .when(platforms: [.macOS])),
            ]
  1. Overriding optimization level by target
    This is very commonly desired. Several times we want to do this, e.g. for dependencies of a Command plugin so that the executable dependency will be built optimised always when used by consumers of the package (I view the command plugin and related dependencies as part of the toolchain, just as I want my swift compiler to have been built with optimization turned on, I want the helper tool for the command plugin that processes tons of json to be optimised for a 2-3x speedup of post processing) - we have run into this in various settings, not just in this case (sometimes a specific part of the code will make debug runs so slow to be unusable, so it makes sense to turn that target to optimized, and conversely, something might need debug stuff).
swiftSettings: [SwiftSetting.unsafeFlags(["-Osize"])]
  1. Enabling experimental features
    (as pointed out by @xwu this now will be in 5.8 thanks to yourself, @Douglas_Gregor and others)
                .unsafeFlags(["-Xfrontend", "-enable-experimental-distributed"]),
            ]
  1. Working around compiler issues
swiftSettings: [
   .unsafeFlags(["-Xfrontend", "-validate-tbd-against-ir=none"]), // due to https://bugs.swift.org/browse/SR-15629
 ]
1 Like

Upcoming features are going to be enabled; those are not experimental. Experimental features aren’t intended to be part of that.

Thanks for the clarification @xwu, I guess then it could be kept as a point of something that is needed and not solved by the upcoming-feature support, as sometime you want to enable an experimental feature even in a package that is versioned after 1.0 (like in our case, distributed actors).