RFC: Tools version with version-specific manifests

SE-0135 introduced a feature that allows a package to supply multiple manifest files by specifying a Swift marketing version. A nice thing about this feature is that it allows packages to adopt newer versions of PackageDescription API without breaking while maintaining compatibility with older tools. However, this feature was added before the tools version feature and its current behavior has a drawback which sometimes breaks working packages when a new Swift version comes out.

Consider a package that contains the following manifests:

Package.swift (tools version 3.0)
Package@swift-4.swift (tools version 4.0)
Package@swift-4.2.swift (tools version 4.2)

SwiftPM picks Package.swift on Swift 3, Package@swift-4.swift on Swift 4, Package@swift-4.2.swift on Swift 4.2 but fallbacks to Package.swift on Swift 5. This is because the other two manifests are version-specific to Swift 4.x and 4.2. Moreover, if there was a Swift 4.3, SwiftPM would pick Package@swift-4.swift because Package@swift-4.2.swift doesn't match that version.

It feels like the correct selection should be Package@swift-4.2.swift on both Swift 5 and Swift 4.3.

Now, consider this case:

Package.swift (tools version 5.0)
Package@swift-4.swift (tools version 4.0)
Package@swift-4.2.swift (tools version 4.2)

SwiftPM picks Package@swift-4.swift on Swift 4, Package@swift-4.2.swift on Swift 4.2 and Package.swift on Swift 5. So far so good but on Swift 4.3, SwiftPM would again pick Package@swift-4.swift and not Package@swift-4.2.swift as one might expect.

I think this problem can be solved by adding an additional behavior if the current swift version doesn't match any of the version-specific manifests:

  • Pick two manifests: the regular Package.swift and the version-specific manifest with a version less than the current swift version.
  • Among these manifests, pick the one which contains a more recent compatible tools version.
3 Likes

This is a good improvement. Attention should also be paid to the interaction between version-specific manifests and the swift package tools-version command.
I haven't been able to use both of them together, so that I still have a janky conditionally-compiled package manifest for my continuous integration. Maybe the reason I tried to use both of them together was the behaviour that's being fixed here, but I didn't keep copious notes.

I think this sounds good in general, but I'd want the behaviour to be less arbitrary than "pick two" and "version less". How about picking among the regular "Package.swift" as well as all version-specific manifests with a version that is supported by current tools. This should always result in the same outcome as the "pick two" rule, but we don't need the "version less" rule.

1 Like

We can ditch the "pick two" rule and just scan all manifests for a compatible tools version but we still need the "less than current version" rule because if a higher Swift version manifests shouldn't be parsed with older tools version. For example:

Package.swift (tools version 5.0)
Package@swift-4.swift (tools version 4.0)
Package@swift-4.2.swift (tools version 4.0)

Without the "less than version" rule, Swift version 4.0 would pick the 4.2 manifest which is wrong even if the tools version 4.0 is compatible.

I think I wasn't clear enough. I didn't mean we should pick all manifests which have compatible tools-version in them, but all which have a compatible version as by SE-0135. With that rule we wouldn't pick the 4.2 one in your example unless I am missing something.

Ah. That works for this example but not for the first example I mentioned in my original post.

You're right. I misread the "a" as a "one" in your original post and also wrongly assumed that the version in the version-specific manifest is language version, but it is the marketing version. So your original proposal is actually good already :+1: Sorry for the confusion.

I like the idea of tweaking the behavior to better support backwards compatibility interaction with the tools version. That said, I'm not crazy about people using this mechanism to support newer Swift versions than what the default Package.swift specifies. That seems like an antipattern that will encourage people to have a confusing set of package manifests. I'd prefer than the pattern we're supporting is "Package.swift is always the newest manifest, and version-specific manifests can be added for backwards compatibility". That was the original intended use for this feature, and just allowing that seems to encourage simpler, clearer package manifest organization.

If that's what we'd want to support, the behavior tweak should probably be:

  • Always use the regular Package.swift manifest unless the tools version is not compatible with the current tools. If not, look for the most recent version-specific manifest with a compatible tools version.
2 Likes

This can lead to an unexpected result (for some users) if the regular Package.swift contains a compatible tools version but there is also a newer version-specific manifest file. Consider these manifests:

Package.swift (tools version 4.0)
Package@swift-4.2.swift (tools version 4.2)

On Swift 4.3, SwiftPM would pick Package.swift but you probably expected Package@swift-4.2.swift.


One possible simple alternative is diagnosing and producing a warning/error if there are version specific manifests that are newer than the current Package.swift. This will stop the spreading of the anti-pattern but the existing packages will require updating. This would be a little unfortunate but maybe that's not so bad considering SwiftPM can avoid introducing a potential difficult to understand/explain behavior.

I think it would be unfortunate to break backwards compatibility. Maybe we could error going forward, e.g. if at least one of the manifests has tools-version 5.0 or later, we would error if we see the anti-pattern but for older tools-versions we implement your original suggestion. That's quite a complicated behaviour, but would avoid breaking existing packages.

I was thinking about that but the points raised by @rballard got me thinking if we should really support this anti-pattern at all. It seems like a lot of special case handling for supporting an anti-pattern. Implementing any kind of handling will cause packages to increase their reliability on the anti-pattern case. I think the proposed behavior is kind of difficult to explain/learn and I am not convinced that it is worth it. The biggest point is that users can do what they really want without SwiftPM providing special handling for this case. Packages that currently break because of this just require a file rename with a corresponding patch release.

I think a warning for tools version < 5 and hard error for tools version >= 5 would be the best approach.

Do we even need a warning? With the current implementation the anti-pattern breaks for everyone who was using it prior to Swift 5, so it could just be a hard error for everyone, couldn't it?

It doesn't necessarily break. It would work if the tools version in the regular Package.swift is supported but is lower than version specific one. Example:

Package.swift (tools version 4.0)
Package@swift-4.2.swift (tools version 4.2)

On Swift 5.0, you would get Package.swift which is supported and should be successfully parsed.

1 Like

I tend to agree with @rballard
It was always by assumption that the main Package.swift would be the default, with possible version specific fallback.

If a project is well maintained, the workflow could indeed be (if it makes sense to keep compliance):

  1. When using a new feature in Package.swift, copy the file with a Swift tool version specific name to freeze that version.
  2. Update the main package version number to start using the new features.

This approach has the added benefit to promote the use of the smallest version number that can possibly work. We are not supposed to force usage of the latest Swift tool if we do not explicitly need it.

I might be talking about something else entirely, but if I have the manifests
Package.swift (tools version 4.2)
Package@swift-4.swift (tools version 4.0)
Package@swift-3.swift (tools version 3.1)

How do I test functionality with Swift 3.2? (ensuring that my library still works when used by a package that is built using a compatibility mode.)
Of course, same applies for future compatibility modes.

Isn't the one catch that SwiftPM didn't always look for the special named Package.swift files and/or look for the comment in the file? i.e. - Package.swift used to have to be left in a state where it support that oldest version, meaning all new things always had to go into a Package@swift-#.swift file?

Maybe by now folks don't care enough about those older versions, but if someone tried, they probably get a cryptic error when that SwiftPM couldn't deal with the contents of the Package.swift.

You are right, SwiftPM didn't yet have the tools-version concept at the time the version-specific manifest files were introduced.

There is currently no flag to specify the manifest that should be used but we should have one (can you file a JIRA for that?). You can either rename the manifest file or use an older toolchain.

@Aciid: [SR-9382] Allow selection of a specific versioned package manifest · Issue #4742 · apple/swift-package-manager · GitHub

1 Like