Language downgrade tool for source libraries

Moderator note: the discussion here was initially in the review thread for lightweight same-type constraints. It picks up from a post by Karoy Lorentey that I was unwilling to move, since it was intended as review feedback.

The idea here is that you write code in the new style and then run the code through a preprocessor before distributing it to clients who aren't using tools that understand it. I don't think this approach precludes any of the modern software engineering practices you mention. If anything, it should reduce the overhead for the library developer by reducing the need to manually maintain #ifs.

5 Likes

That sounds like a great idea.

(I think such a tool could just ship in the standard toolchain; I don't see why it would need to be a third party thing.)

5 Likes

Maintaining a post-processing workflow isn’t free of engineering cost either. For example, the typical localization process on Apple platforms involves running genstrings periodically to collect localizable strings into .strings files. In addition to having to double check that genstrings correctly captured all the localizable strings, the fact that this is a manual process leads it to be deferred, which delays discovering localization issues (e.g. a single localizable string needs to be broken into multiple variants to support a target language’s grammar).

Requiring library authors to run a tool to munge their code into a backward-compatible variant increases each library author’s persistent risk of discovering late-breaking backward compatibility issue.

This is a great idea: do we have any more detail on it? I can see some issues with integrating with SwiftPM, most notably that as long as we rely on git clone to distribute source-available libraries then "distribute" is not really a discrete step. You end up maintaining parallel branches, one for each Swift release you support, which you'll presumably have to automate cross-merges to. I'm not sure this idea is totally straightforward.

While I'm here I'll also say that while this idea is a good one, that tool does not exist yet, so in the meantime the pain remains.

2 Likes

Could the #if#endif be inside a multiline comment, in such a way that new compiler versions would still recognize it? For example:

@available(SwiftStdlib 5.1, *)
public protocol Identifiable

/*$$
 #if compiler(>=5.7) && $PrimaryAssociatedTypes
 <ID>
 #endif
 $$*/

{ // The body of the protocol is unchanged.

  /// A type representing the stable identity of the entity associated with
  /// an instance.
  associatedtype ID: Hashable

  /// The stable identity of the entity associated with this instance.
  var id: ID { get }
}

Maybe it's a great idea, but there are three caveats:

  1. With such a reasoning, it is the community, whoever it is, who is responsible for providing such tools.

    Until then, library authors can not deal with multiple language versions. They'll have to choose between postponing support for new language features, or dropping support for older languages.

    On top of that, the people who need the tools may not the people who are able to make it. This means that 1. nothing guarantees that such a tool would become public, and 2. a public tool, built by one particular team, may not fit the needs of other teams. Building general tools, in the context of library distribution, is hard - especially considering that SPM is not the only way to distribute code (more on that below), and that Swift targets Darwin, Linux, and Windows.

  2. The problem is addressed with a build step.

    I do not think it is reasonable to assume that build skills are as frequent as developing skills. Building is much more difficult that developing. We need a welcoming environment for developers to ship libraries, and developers to use libraries. Adding more build steps in order to be able to maintain or use a library over time is a step in the wrong direction. We need less build steps.

  3. The problem is addressed with an unclear distribution scheme.

    Please elaborate on what you mean with "run the code through a preprocessor before distributing it to clients who aren't using tools that understand it". I do not know how to distribute different code bases for different compiler versions with the same version tag. Please consider that a library author may want to distribute over SPM, but also CocoaPods, maybe Carthage, as well as support direct including of an Xcode project. I surely have forgotten other distribution schemes.

9 Likes

This is a really complex way of avoiding simply recognizing that some form of lexically-scoped if is a necessary tool for long-term library maintenance in the presence of source and binary compatibility requirements.

(I mean, it is a lexically-scoped if, but in trying to pretend that it isn't it becomes a lot fussier than it should be.)

2 Likes

This isn’t a thoroughly fleshed-out idea, but my hope would be that such a tool would produce code with the #ifs necessary to be interpreted by any version of the tools. So one possible development approach would be:

  • You do your development in branches normally.
  • When you want to make a release, you cut a new branch, have the downgrade tool rewrite your source in place, and then tag a commit on that branch as a release.

I believe this is consistent with most package managers, which want to check out specific tags. It shouldn’t require a client-side build step or maintaining different branches for different tools versions.

Such tools are only useful for new syntactic sugars, and most of the new features are still not available. Source-generation is not reliable without additional feature validation from the compiler.

I have used similar tools before like Rector for PHP and SwiftFormat for Swift. Both of them are rule-based and perform transformation directly on the working tree. In a real world project written with a newer version, it is very likely to depend on some features that require additional backporting (Swift term) or polyfill (PHP term). And, if there’s no compiler-level backport validation support, it is still impossible for developers to know if their codes really work on an older version without CI testing.

That is, we’re likely to have no more reliable solution than what SwiftFormat already provides. If anyone want such tool officially, it’s better to empower swift-format with such functionality, instead of making it into the compiler or SwiftPM.

I don't really agree with this. The purpose of swift-format is to provide code style diagnostics and auto-corrections; we shouldn't add unrelated functionality for library-source-downgrading simply because swift-format also happens to be a tool that operates on the syntax tree of the source code.

It's better to have separate tools each focused on their specific, intended purpose instead of going with a "kitchen sink" approach. Since swift-syntax lets anyone write tooling that operates on syntax trees, then if that were the method that was chosen to implement it, there's not a very compelling reason for the tooling being discussed in this thread to be part of a code style formatter.

4 Likes

I don't see how a preprocessor tool could help with anything but trivial syntactic sugar improvements, and I would much rather simply not use new syntactic sugar if the cost is having to package releases which have different code than what we're actually writing and testing.

#if being able to appear in more places in the syntax would do a lot more to ease the pain of supporting multiple Swift versions.

Yes, it would be very limited, basically to things that can be simply be removed for clients with old tools. And some of those things really ought to be removable with a very specific #if, like attributes, although unfortunately they currently aren’t. It’s just an idea for how we can help people in this situation without completely hamstringing the evolution of the language.

I’m not sure that the testing argument is very strong, though. If you’re claiming that your library works with older tools, you really ought to be testing that periodically, and nothing about a downgrade tool will stop you from doing that.

I want to emphasize the difference between a tool-based approach and inline conditionals.

Inline conditionals live in my source code. While I’m viewing the source, I can see exactly how the source file handles backporting to older clients. I test against the same source file with older and newer toolchains.

With a post-processing tool, I have to track a separate artifact. That artifact is hidden from me while I’m viewing or changing the authoritative source. My testing burden extends not only to ensuring the backward compatibility of the source, but also the behavior of the tool itself. And of course there are all the issues others have mentioned about the impedance mismatch between a tool-based approach and package distribution.

Swift already has inline conditionals. We are discussing the addition of a post-processing tool as a critical component of back-deployment workflow, thus doubling the number of back-deployment techniques that library authors must adopt.

I implore the core team to reconsider their understanding of the importance of inline conditionals when designing back-deployment of language features.

5 Likes

If people aren't interested in using a tool like this, that's useful feedback. The most likely result, though, is that you will simply continue to not be able to adopt some features until you're willing to bump your language version.

Has the core team surveyed how other languages handle situations like this? It’s one thing to say a language feature doesn’t back-deploy; it’s another to say that codebases which back-deploy can’t conditionally adopt features when not back-deploying.

You can conditionally adopt these features. You will have to use #if. You are unwilling to use a tool that does that for you, so you will need to use #if manually. We can make #if more flexible, but that will also not back-deploy, so you will not be able to take advantage of it. That is the situation.

3 Likes

Good point.

2 Likes

I guess it depends on what time horizon you’re considering. It could possibly help a future self to backport from swift 7.y to 6.x.

3 Likes

as a library author, one outcome i forsee is that people will adapt to this by cutting off support at Swift 5.7, just like a lot of us are currently positioning the cutoff at Swift 5.3. the reason for this is that it’s pretty easy to backport Swift 5.6 libraries to support 5.5, 5.4, and 5.3, but it’s comparatively difficult to backport 5.3 to 5.2.

right now, i think that the swift community benefits immensely from the fact that we have a window of three past minor releases that the majority of libraries can support with minimal effort. i imagine there are a lot of complaints, issue reports, and questions we are not getting because libraries “just work” with any swift toolchain from the past year and a half.

2 Likes

Obviously #if improvements can't help with adopting the feature that prompted this, but it's also something that wouldn't make sense to #if even if you could. I think it sounds like a great change that I look forward to being able to use, but writing both the sugared and non-sugared version would just be silly.

Others probably disagree with this, but I'm generally completely fine with the idea that we have to wait a year to adopt new Swift features that make writing Swift more pleasant. That's just a natural result of supporting year-old versions of Swift, and trying to avoid that doesn't seem worth it (and I'm happy it's not the 3+ year delay for C++ features...). Where it's frustrating is when providing the best API to our users possible without breaking backwards compatibility requires copying and pasting significant amounts of code. #if being unable to guard an attribute on a function without duplicating the function is by far the most common culprit for this.

I think "we are uncomfortable with accepting a restriction that all declaration syntax must be designed around the ease of #if ing it." is a very reasonable position, as an obvious possible result of that restriction would be that five years down the road the language's syntax makes no sense because every decision was made to optimize for something relevant years ago but not relevant now. I would like though if changes which are not purely syntactic sugar which are difficult to conditionally adopt also came with some time spent on solving the reason why it's difficult to conditionally adopt so that future changes aren't awkward for the same reason.

6 Likes