Swift Async Algorithms 1.1 Planning

[Philippe_Hausler]

Swift Async Algorithms started off with a goal for source stability. This means that building against newer versions should not break older code using it.

Since we signed up for semantic versioning no public interface shall be broken (and the only potential exception to this is a transition of major versions). However we do want to ensure that versioning is meaningful and follows a cadence of regular releases. Tying releases to a specific chronological guarantee is not nearly as meaningful since it is not tied to a binary release of an operating system nor is it tied to any specific cadence of releases of hardware. Instead it makes more sense to build semantic release cadences around feature sets that indicate the start and end of periods of work around certain ideas.

The 1.0 release was built around getting a set of ideas of the fundamental algorithms to build a framework that is cohesive. This meant expanding upon the AsyncSequence types provided in the standard library (specifically the _Concurrency module) for the ones that would still be commonly needed but not the most fundamental expected for any use. Part of the focus was that of interfacing with clocks and durations, others were focused upon buffering and splitting.

Since we have a released semantic version we need to demarcate the start of a new release for development. Swift modules thankfully have the capability of being specified of a certain branch or release and can specify a valid range for use. This means that developers may be able to specify the range of usage as 1.0 or allow 1.0 and beyond or require 1.1 or later. It is expected that any cut release version should be self contained and usable within that version. That means our qualification of which should ensure that any start of development for 1.1 should contain stable development for landing that released version. Since any new development will be on the main branch, any minor fixes that need to be applied to the 1.0.4 release would then be cherry picked (if needed) and the version of the release would then be bumped to 1.0.5. It is also expected that those fixes would be present on the main branch such that they are available for the eventual 1.1.

So at this point the question is: what constitutes 1.1? Since the idea is not to be bound to a yearly cadence (either shorter or longer than year) but instead bound to a conceptual grouping - 1.1 will have a conceptual grouping of adding in functionality to replace existing non-Swift-Concurrency systems with Swift Concurrency.

So far there are a few important algorithms to bring forward for this release. These all have similar systems that exist in other frameworks or are intrinsically linked to interfacing with non-Swift-Concurrency systems.

• share()
• MultiProducerSingleConsumerChannel
• enumerated()
• deferred()
• withDeadline
• withLatestFrom
• mapFailure
• throttle (this is just a revisit of behaviors)

In addition to these areas of focus there are a few language improvements that are worth considering as well.

• variadic generics (N-ary zip et al)
• adopting some types (by deprecating old initializers and adding new ones with typed throws)
• typed throws

Of course there is a chance where not all of these will make it; for 1.0 throttle was deferred from the release due to complications with the behaviors. However, any deferred algorithm from 1.1 will be something of strong consideration for 1.2.

AsyncAlgorithms is not my sole responsibility so I will need some help from the community to make this deliverable happen. Some folks have already posted patches for the algorithms and features listed. In order to move them forward the major things that need to happen is first and foremost communication around what is and is not finished (the other requirements are somewhat flexible and time permitting can be relaxed), the other (more flexible requirements) are a write-up of a proposal, an implementation, and tests covering at least a good swath of the behaviors and code coverage. In the coming days I will be either promoting bugs or writing up issues for a 1.1 milestone.

This general process is something that I would also like to refine and perhaps advocate to other packages like swift-algorithms or numerics etc. Some of this process was inspired from some insight from @nnnnnnnn from ArgumentParser.

I would love to hear what aspects of this quasi-pitch should be refined or changed to better suit the needs of the community.

[vanvoorden (Rick van Voorden)]

@Philippe_Hausler

variadic generics (N-ary zip et al)

I believe there were some ideas and snippets here that might give an engineer a head start on that:

Is it possible to create a cartesian product function with parameter packs?

The shape of the function would be func variableCartesianProduct<each T>(input: repeat [each T]) -> [(repeat each T)] The use case here is as input for parameterized tests. I'd like to take lists of possible inputs and produce all combinations of those lists. I've made a few attempts, but I think there are some missing pieces. You could build up recursive tuples, but I can't figure out a way to flatten them. ie you can't do a transformation of an arbitrary tuple shaped like ((String, Int), Flo…

[ph1ps (Philipp Gabriel)]

@Philippe_Hausler

So far there are a few important algorithms to bring forward for this release. These all have similar systems that exist in other frameworks or are intrinsically linked to interfacing with non-Swift-Concurrency systems.

• share()
• MultiProducerSingleConsumerChannel
• enumerated()
• deferred()
• withDeadline
• withLatestFrom
• mapFailure
• throttle (this is just a revisit of behaviors)

Is retry something AsyncAlgorithms would be interested in? Certainly not the most complicated algorithm implementation-wise, but I still had a need for it in the past, and Combine for instance has it.

Besides that, is splitting up the algorithms into submodules something that was considered? swift-collections has this, although swift-algorithms does not, so there is no preferred way I guess. But I mostly only use 1-2 algorithms when I depend on AsyncAlgorithms myself.

[vanvoorden (Rick van Voorden)]

@ph1ps

Besides that, is splitting up the algorithms into submodules something that was considered? swift-collections has this, although swift-algorithms does not, so there is no preferred way I guess. But I mostly only use 1-2 algorithms when I depend on AsyncAlgorithms myself.

I believe the solution being considered is package traits:

@FranzBusch

@vanvoorden

Maybe this is more of a feedback for the diff review… but do we know what kind of horizontal dependencies we expect between share and the rest of the package? Could this potentially ship in a "subpackage" target under AsyncAlgorithms so that a product engineer that needs only share could link just against that code and not pay a binary size and compile time penalty for code they might not need?

This has come up before and we could leverage package traits to move each algorithm behind a trait and enable all by default. This would allow other packages to only pick the algorithms they need.

[Jon_Shier]

Unless I'm missing something, there's really no need to overthink this. Semantic versioning with SPM is essentially zero cost, so there's no reason not to release as often as you can. This is especially true of bug fixes, but new features are appreciated by the community as quickly as you can produce them.

@vanvoorden

I believe the solution being considered is package traits:

Since Xcode doesn't support package traits, this doesn't really help most Swift developers. Sure, you can work around that limitation by using a local package for those dependencies, but that's a non-obvious solution that most won't see and so won't see the features hidden by the traits.

[Philippe_Hausler]

@ph1ps

Is retry something AsyncAlgorithms would be interested in?

Yep that would definitely fit - anything in that category of "needed it but Combine had it" would be under the banner of "replacements for existing non-Swift-Concurrency".

@ph1ps

Besides that, is splitting up the algorithms into submodules something that was considered?

This one I am hesitant on unless strongly motivated by a compelling reason. AsyncAlgorithms isn't a huge module binary cost wise so that really doesn't justify to me the maintenance and development burden.

@Jon_Shier

Since Xcode doesn't support package traits, this doesn't really help most Swift developers.

This is perhaps the most compelling reasoning why not to focus on that for this cycle.

@Jon_Shier

Semantic versioning with SPM is essentially zero cost, so there's no reason not to release as often as you can.

True, and I don't want to tie things down to a yearly cadence requirement; if we can get things together before that timeframe to a point that seems satisfactory then bumping the version seems like the right move to me. Per small fixes - I think as those come up I am willing to cut minor update releases as things arise. Usually I just need someone to ask and I think cutting those is not a huge deal.

[hassila (Joakim Hassila)]

Very happy to see this direction - just to clarify, the throttling semantics is what we previously discussed in:

Surprising semantics of throttled sequences

After some experimenting today after seeing unexpected behavior I just realized that the throttled async stream behaves in a quite unexpected manner. Basically the current semantics easily leads to lost Updates/wakeups. AFAICT the current semantics are that if a new event comes in within the specified throttled time on the base async sequence, it’s not forwarded. But after the specified throttled time has elapsed- it’s not forwarded either until a new event comes from the base sequence. So ba…

Or are there something else in addition that I missed?

[ibex10]

@Philippe_Hausler

• MultiProducerSingleConsumerChannel

My apologies for asking this dumb question, but why is MultiProducerSingleConsumerChannel considered an algorithm?

Isn't it just a data structure, which an algorithm might use to solve a problem? For example, a sorting algorithm might use an Array, but can an Array be called an algorithm?

[Agent (Donovan Voss)]

I’m interested in a sort of fork-map thing, where:

Backbone async sequence of A elements
╌╌╌ ● ╌╌╌╌ ● ╌╌╌ ● ╌╌╌╌
↘︎ (map to async sequence of Aʹ)
  ╌╌╌ ■ ╌╌╌╌ ■ ╌╌╌ ■ ╌╌╌╌ ⇒ consumer of Aʹ
↘︎ (map to async sequence of Aʺ)      
  ╌╌╌ ◆ ╌╌╌╌ ◆ ╌╌╌ ◆ ╌╌╌╌ ⇒ consumer of Aʺ

where a base element can be wrapped, recast, and processed by several different consumers.

Is this what share() would help with, or does this need a different solution?

[Philippe_Hausler]

That would be something like

let shared = base.share()
let a = shared.map { ... }
let b = shared.map { ... }

[DevAndArtist (Adrian Meister)]

I‘m no longer actively using Swift nowadays, however I wouldn’t mind if APIs were introduced to at least allow opting out of the eager cancellation behavior of the stream family types.

AsyncSequences and cooperative task cancellation

In this thread I would like to discuss the cooperative task cancellation in context of AsyncSequence conforming types. Spoiler ahead, I personally think that some of the exiting async sequences might have an unwanted behavior and are not cooperative, at least from my point of view. With this feedback I would like to spark a discussion regarding how we should evaluate proposed async sequence algorithms and that we should check the existing types which already landed with _Concurrency module. I p…

This is long overdue imo.

[ph1ps (Philipp Gabriel)]

Is a migration from XCTest to Swift Testing also something that is being / should be tackled? I know that this is not relevant for any release, but I guess for writing tests for new algorithms.

I checked AsyncAlgorithms_XCTest and it seemed like it can easily be converted (and AsyncSequenceValidation is test system agnostic), the rest also does not seem like undoable task in terms of complexity.

Edit: I guess the performance tests are not yet possible.

[Philippe_Hausler]

Migrating is a reasonable task at any point in time and might provide useful feedback for the Swift Testing proposals / pitches that folks might be interested in pushing.

[ph1ps (Philipp Gabriel)]

@Philippe_Hausler

This one I am hesitant on unless strongly motivated by a compelling reason. AsyncAlgorithms isn't a huge module binary cost wise so that really doesn't justify to me the maintenance and development burden.

My intial thought was, that not every algorithm requires swift-collections, but I guess the binary size impact of that is also minimal.

Something I‘ve also thought about recently is, if we could introduce an Evolution.md (similar to what swift-foundation has) to swift-async-algorithms. I couldn‘t say I‘m 100% certain on how evolution works in this project.