SE-0298: Async/Await: Sequences

Combine is for events being pushed. You are not in control of when the events are fired. AsyncSequence is for reading a stream asynchronously. Typically used where there is IO involved and you don’t want to block a thread while waiting for the next chunk of data.

Important improvement is readability, you no longer need to translate imperative code to a declarative pipeline of Combine operators and to manage subscriptions. This is especially handy for long pipelines that contain operators like flatMap and frequently require eraseToAnyPublisher().

for await x in someAsyncSequence {
  let y = transform(x)
  guard let z = doSomething(y) else { continue }

  let a: A
  if branch(z) {
    a = await doSomethingAsync(z)
  } else {
    a = doSomethingElse(z)
  }

  guard check(a) else { throw CheckFailed() }

  print(a)
}

Compare this with the Combine version:

var subscriptions = [AnyCancellable]()
someAsyncSequence
  .map(transform)
  .compactMap(doSomething)
  .flatMap { z -> AnyPublisher<A, Error> in
    if branch(z) {
      return doSomethingAsync(z).eraseToAnyPublisher()
    } else {
      return Just(doSomethingElse(z)).eraseToAnyPublisher()
    }
  }.tryMap { a in
    guard check(a) else { throw CheckFailed() }
    return a
  }.sink { a in
    print(a)
  }.store(in: &subscriptions)

In the latter case not only you're forced to use eraseToAnyPublisher() and to provide an explicit type signature, the code is much harder to read because of the braces clutter. This is especially hard when teaching beginners, I look forward to explaning await just once instead of explaining all of the necessary advanced topics required by Combine, like type erasure, subscriptions etc.

Most importantly, memory management with Combine is much more tricky. Since flatMap, handleEvents, and sink are inherently "imperative" operators, it's common to capture a reference to some state outside of their respective closures (say auth tokens for networking calls etc). It's so much easier to create an unwanted memory cycle that way. With async/await these issues are nipped in the bud by avoiding unnecessary closure scopes.

10 Likes

I'm not sure this is a fair comparison. When given an arbitrary Publisher you also may not be in control of when the events are fired. OTOH nothing prevents you from creating a custom AsyncSequence that allows you to "publish" values on it manually, even from another thread if needed.

Likewise, Combine publishers are just as useful when IO is involved and you don't want block a thread, as you can subscribe publishers on an appropriate non-blocking scheduler.

4 Likes
  • What is your evaluation of the proposal?
    +1 iterating is a basic activity.

Not including it would be a significant miss.

  • Is the problem being addressed significant enough to warrant a change to Swift?

Almost all work involves some level of iteration. Yes.

  • Does this proposal fit well with the feel and direction of Swift?

The language is familiar. Although it should be clearly documented if each item is run parallel and all are waited for vs. each iteration waiting to complete before the next item.

  • If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?

NodeJS is not blocking by default and requires some getting used to. Being clear about “on / off” is important for reduced “cognitive load” of whoever is programming. If I think about it from a test If perspective, I want to know how I would be able to write tests against this. Going forward it should be clear about how to write tests against feature like this.

  • How much effort did you put into your review? A glance, a quick reading, or an in-depth study?

A quick reading.

I'd love to do some concurrency tests using this toolchain, but for some reason both it and all other nightly builds since December error out with all but the most trivial projects.

Is it appropriate to report bugs against nightly toolchains or do you have an idea how I could proceed in generating a toolchain that bases on the most stable version (5.3) and only incorporates the concurrency part? For reference, here's the crash I'm getting:

CompileSwift normal x86_64 /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/ItemSlider/ItemSliderViewController.swift (in target 'MediaBeam' from project 'MediaBeam')
    cd /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam
    /Library/Developer/Toolchains/swift-PR-35224-820.xctoolchain/usr/bin/swift-frontend -frontend -c /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/ItemDetails/ImageViewController.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/App\ \&\ Misc/TableSectionHeaderView.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/Backend/ImageLoader.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/MediaPods/DirectoryViewController.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/Backend/Models/IOTToken.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/MediaPods/DirectoryEntryTableViewCell.swift -primary-file /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/ItemSlider/ItemSliderViewController.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/Backend/Session.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/MediaPods/MediaPodTableViewCell.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/ClaimToken/ClaimTokenViewController.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/MediaPods/MediaPodsViewController.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/Backend/Models/MediaPod.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/Backend/Models/Credentials.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/Menu/MenuViewController.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/Backend/WebDAV.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/App\ \&\ Misc/AppDelegate.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/Backend/REST.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/Backend/Models/IOTTokenIssuer.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/App\ \&\ Misc/SceneDelegate.swift -emit-module-path /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/Objects-normal/x86_64/ItemSliderViewController\~partial.swiftmodule -emit-module-doc-path /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/Objects-normal/x86_64/ItemSliderViewController\~partial.swiftdoc -emit-module-source-info-path /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/Objects-normal/x86_64/ItemSliderViewController\~partial.swiftsourceinfo -serialize-diagnostics-path /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/Objects-normal/x86_64/ItemSliderViewController.dia -emit-dependencies-path /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/Objects-normal/x86_64/ItemSliderViewController.d -emit-reference-dependencies-path /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/Objects-normal/x86_64/ItemSliderViewController.swiftdeps -target x86_64-apple-ios13.0-simulator -enable-objc-interop -sdk /Applications/Xcode-12.3.0.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator14.3.sdk -I /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Products/Debug-iphonesimulator -F /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Products/Debug-iphonesimulator -enable-testing -g -module-cache-path /Users/mickey/Library/Developer/Xcode/DerivedData/ModuleCache.noindex -swift-version 5 -enforce-exclusivity\=checked -Onone -D DEBUG -serialize-debugging-options -enable-anonymous-context-mangled-names -Xcc -fmodule-map-file\=/Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/GeneratedModuleMaps-iphonesimulator/CCryptoBoringSSLShims.modulemap -Xcc -fmodule-map-file\=/Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/GeneratedModuleMaps-iphonesimulator/CCryptoBoringSSL.modulemap -Xcc -fmodule-map-file\=/Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/GeneratedModuleMaps-iphonesimulator/YapDatabase.modulemap -Xcc -I/Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/swift-overrides.hmap -Xcc -iquote -Xcc /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/MediaBeam-generated-files.hmap -Xcc -I/Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/MediaBeam-own-target-headers.hmap -Xcc -I/Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/MediaBeam-all-non-framework-target-headers.hmap -Xcc -ivfsoverlay -Xcc /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/all-product-headers.yaml -Xcc -iquote -Xcc /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/MediaBeam-project-headers.hmap -Xcc -I/Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/SourcePackages/checkouts/swift-crypto/Sources/CCryptoBoringSSLShims/include -Xcc -I/Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/SourcePackages/checkouts/swift-crypto/Sources/CCryptoBoringSSL/include -Xcc -I/Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/SourcePackages/checkouts/SwiftYapDatabase/Sources/YapDatabase/include -Xcc -I/Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Products/Debug-iphonesimulator/include -Xcc -I/Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/DerivedSources-normal/x86_64 -Xcc -I/Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/DerivedSources/x86_64 -Xcc -I/Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/DerivedSources -Xcc -DDEBUG\=1 -Xcc -working-directory/Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam -target-sdk-version 14.3 -module-name MediaBeam -o /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/Objects-normal/x86_64/ItemSliderViewController.o -index-store-path /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Index/DataStore -index-system-modules

/Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/ItemSlider/ItemSliderViewController.swift:74:47: warning: string interpolation produces a debug description for an optional value; did you mean to make this explicit?
            print("huh? viewControllers are \(self.viewControllers)")
                                              ^~~~~~~~~~~~~~~~~~~~
/Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/ItemSlider/ItemSliderViewController.swift:74:52: note: use 'String(describing:)' to silence this warning
            print("huh? viewControllers are \(self.viewControllers)")
                                              ~~~~~^~~~~~~~~~~~~~~
                                              String(describing:  )
/Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/ItemSlider/ItemSliderViewController.swift:74:52: note: provide a default value to avoid this warning
            print("huh? viewControllers are \(self.viewControllers)")
                                              ~~~~~^~~~~~~~~~~~~~~
                                                                   ?? <#default value#>
/Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/Backend/Session.swift:66:46: warning: immutable value 'error' was never used; consider replacing with '_' or removing it
                    case .failure(error: let error):
                                         ~~~~^~~~~
                                         _
/Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/Backend/Session.swift:68:45: warning: immutable value 'status' was never used; consider replacing with '_' or removing it
                    case .empty(status: let status):
                                        ~~~~^~~~~~
                                        _
/Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/Backend/Session.swift:70:47: warning: immutable value 'status' was never used; consider replacing with '_' or removing it
                    case .success(status: let status, payload: let payload):
                                          ~~~~^~~~~~
                                          _
/Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/Backend/Session.swift:92:42: warning: immutable value 'error' was never used; consider replacing with '_' or removing it
                case .failure(error: let error):
                                     ~~~~^~~~~
                                     _
/Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/Backend/Session.swift:94:41: warning: immutable value 'status' was never used; consider replacing with '_' or removing it
                case .empty(status: let status):
                                    ~~~~^~~~~~
                                    _
/Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/Backend/Session.swift:96:43: warning: immutable value 'status' was never used; consider replacing with '_' or removing it
                case .success(status: let status, payload: let payload):
                                      ~~~~^~~~~~
                                      _
Assertion failed: (isActuallyCanonicalOrNull() && "Forming a CanType out of a non-canonical type!"), function CanType, file /Users/buildnode/jenkins/workspace/swift-PR-toolchain-osx/branch-main/swift/include/swift/AST/Type.h, line 402.
Please submit a bug report (https://swift.org/contributing/#reporting-bugs) and include the project and the crash backtrace.
Stack dump:
0.	Program arguments: /Library/Developer/Toolchains/swift-PR-35224-820.xctoolchain/usr/bin/swift-frontend -frontend -c /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/ItemDetails/ImageViewController.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/App & Misc/TableSectionHeaderView.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/Backend/ImageLoader.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/MediaPods/DirectoryViewController.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/Backend/Models/IOTToken.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/MediaPods/DirectoryEntryTableViewCell.swift -primary-file /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/ItemSlider/ItemSliderViewController.swift -primary-file /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/Backend/Session.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/MediaPods/MediaPodTableViewCell.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/ClaimToken/ClaimTokenViewController.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/MediaPods/MediaPodsViewController.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/Backend/Models/MediaPod.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/Backend/Models/Credentials.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/Menu/MenuViewController.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/Backend/WebDAV.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/App & Misc/AppDelegate.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/Backend/REST.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/Backend/Models/IOTTokenIssuer.swift /Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/App & Misc/SceneDelegate.swift -emit-module-path /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/Objects-normal/x86_64/ItemSliderViewController~partial.swiftmodule -emit-module-path /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/Objects-normal/x86_64/Session~partial.swiftmodule -emit-module-doc-path /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/Objects-normal/x86_64/ItemSliderViewController~partial.swiftdoc -emit-module-doc-path /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/Objects-normal/x86_64/Session~partial.swiftdoc -emit-module-source-info-path /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/Objects-normal/x86_64/ItemSliderViewController~partial.swiftsourceinfo -emit-module-source-info-path /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/Objects-normal/x86_64/Session~partial.swiftsourceinfo -serialize-diagnostics-path /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/Objects-normal/x86_64/ItemSliderViewController.dia -serialize-diagnostics-path /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/Objects-normal/x86_64/Session.dia -emit-dependencies-path /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/Objects-normal/x86_64/ItemSliderViewController.d -emit-dependencies-path /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/Objects-normal/x86_64/Session.d -emit-reference-dependencies-path /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/Objects-normal/x86_64/ItemSliderViewController.swiftdeps -emit-reference-dependencies-path /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/Objects-normal/x86_64/Session.swiftdeps -target x86_64-apple-ios13.0-simulator -enable-objc-interop -sdk /Applications/Xcode-12.3.0.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator14.3.sdk -I /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Products/Debug-iphonesimulator -F /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Products/Debug-iphonesimulator -enable-testing -g -module-cache-path /Users/mickey/Library/Developer/Xcode/DerivedData/ModuleCache.noindex -swift-version 5 -enforce-exclusivity=checked -Onone -D DEBUG -serialize-debugging-options -enable-anonymous-context-mangled-names -Xcc -fmodule-map-file=/Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/GeneratedModuleMaps-iphonesimulator/CCryptoBoringSSLShims.modulemap -Xcc -fmodule-map-file=/Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/GeneratedModuleMaps-iphonesimulator/CCryptoBoringSSL.modulemap -Xcc -fmodule-map-file=/Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/GeneratedModuleMaps-iphonesimulator/YapDatabase.modulemap -Xcc -I/Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/swift-overrides.hmap -Xcc -iquote -Xcc /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/MediaBeam-generated-files.hmap -Xcc -I/Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/MediaBeam-own-target-headers.hmap -Xcc -I/Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/MediaBeam-all-non-framework-target-headers.hmap -Xcc -ivfsoverlay -Xcc /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/all-product-headers.yaml -Xcc -iquote -Xcc /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/MediaBeam-project-headers.hmap -Xcc -I/Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/SourcePackages/checkouts/swift-crypto/Sources/CCryptoBoringSSLShims/include -Xcc -I/Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/SourcePackages/checkouts/swift-crypto/Sources/CCryptoBoringSSL/include -Xcc -I/Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/SourcePackages/checkouts/SwiftYapDatabase/Sources/YapDatabase/include -Xcc -I/Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Products/Debug-iphonesimulator/include -Xcc -I/Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/DerivedSources-normal/x86_64 -Xcc -I/Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/DerivedSources/x86_64 -Xcc -I/Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/DerivedSources -Xcc -DDEBUG=1 -Xcc -working-directory/Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam -target-sdk-version 14.3 -module-name MediaBeam -o /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/Objects-normal/x86_64/ItemSliderViewController.o -o /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Build/Intermediates.noindex/MediaBeam.build/Debug-iphonesimulator/MediaBeam.build/Objects-normal/x86_64/Session.o -index-store-path /Users/mickey/Library/Developer/Xcode/DerivedData/MediaBeam-herrtttfbqaecxcqwvbkhsgejipk/Index/DataStore -index-system-modules 
1.	Apple Swift version 5.3-dev (LLVM 67722b8904ff3f1, Swift a0783a30ada3493)
2.	While evaluating request IRGenRequest(IR Generation for file "/Volumes/Transcend/Documents/late/mediabeam/iOS/MediaBeam/MediaBeam/Backend/Session.swift")
3.	While evaluating request ExecuteSILPipelineRequest(Run pipelines { IRGen Preparation } on SIL for MediaBeam.MediaBeam)
4.	While running pass #129 SILModuleTransform "LoadableByAddress".
0  swift-frontend           0x0000000111e48e25 llvm::sys::PrintStackTrace(llvm::raw_ostream&) + 37
1  swift-frontend           0x0000000111e48085 llvm::sys::RunSignalHandlers() + 85
2  swift-frontend           0x0000000111e493f6 SignalHandler(int) + 262
3  libsystem_platform.dylib 0x00007fff2040dd7d _sigtramp + 29
4  libsystem_platform.dylib 000000000000000000 _sigtramp + 18446603339975041696
5  libsystem_c.dylib        0x00007fff2031c720 abort + 120
6  libsystem_c.dylib        0x00007fff2031b9d6 err + 0
7  swift-frontend           0x00000001121bd063 swift::SILBuilder::createTuple(swift::SILLocation, llvm::ArrayRef<swift::SILValue>) (.cold.1) + 35
8  swift-frontend           0x000000010e15c1fb swift::SILBuilder::createTuple(swift::SILLocation, llvm::ArrayRef<swift::SILValue>) + 507
9  swift-frontend           0x000000010d9c699c (anonymous namespace)::LoadableByAddress::run() + 4364
10 swift-frontend           0x000000010dd9c47e swift::SILPassManager::runModulePass(unsigned int) + 558
11 swift-frontend           0x000000010dda112a swift::SILPassManager::execute() + 666
12 swift-frontend           0x000000010dd99228 swift::SILPassManager::executePassPipelinePlan(swift::SILPassPipelinePlan const&) + 72
13 swift-frontend           0x000000010dd991c3 swift::ExecuteSILPipelineRequest::evaluate(swift::Evaluator&, swift::SILPipelineExecutionDescriptor) const + 51
14 swift-frontend           0x000000010ddbbfbd swift::SimpleRequest<swift::ExecuteSILPipelineRequest, std::__1::tuple<> (swift::SILPipelineExecutionDescriptor), (swift::RequestFlags)1>::evaluateRequest(swift::ExecuteSILPipelineRequest const&, swift::Evaluator&) + 29
15 swift-frontend           0x000000010dda3690 llvm::Expected<swift::ExecuteSILPipelineRequest::OutputType> swift::Evaluator::getResultUncached<swift::ExecuteSILPipelineRequest>(swift::ExecuteSILPipelineRequest const&) + 240
16 swift-frontend           0x000000010dd99462 swift::executePassPipelinePlan(swift::SILModule*, swift::SILPassPipelinePlan const&, bool, swift::irgen::IRGenModule*) + 82
17 swift-frontend           0x000000010d961c7a swift::IRGenRequest::evaluate(swift::Evaluator&, swift::IRGenDescriptor) const + 1850
18 swift-frontend           0x000000010d99ca8d swift::GeneratedModule swift::SimpleRequest<swift::IRGenRequest, swift::GeneratedModule (swift::IRGenDescriptor), (swift::RequestFlags)9>::callDerived<0ul>(swift::Evaluator&, std::__1::integer_sequence<unsigned long, 0ul>) const + 157
19 swift-frontend           0x000000010d99c9ae swift::SimpleRequest<swift::IRGenRequest, swift::GeneratedModule (swift::IRGenDescriptor), (swift::RequestFlags)9>::evaluateRequest(swift::IRGenRequest const&, swift::Evaluator&) + 14
20 swift-frontend           0x000000010d96d938 llvm::Expected<swift::IRGenRequest::OutputType> swift::Evaluator::getResultUncached<swift::IRGenRequest>(swift::IRGenRequest const&) + 408
21 swift-frontend           0x000000010d964859 swift::performIRGeneration(swift::FileUnit*, swift::IRGenOptions const&, swift::TBDGenOptions const&, std::__1::unique_ptr<swift::SILModule, std::__1::default_delete<swift::SILModule> >, llvm::StringRef, swift::PrimarySpecificPaths const&, llvm::StringRef, llvm::GlobalVariable**) + 313
22 swift-frontend           0x000000010d6882b8 performCompileStepsPostSILGen(swift::CompilerInstance&, std::__1::unique_ptr<swift::SILModule, std::__1::default_delete<swift::SILModule> >, llvm::PointerUnion<swift::ModuleDecl*, swift::SourceFile*>, swift::PrimarySpecificPaths const&, int&, swift::FrontendObserver*) + 2504
23 swift-frontend           0x000000010d6875ae performCompileStepsPostSema(swift::CompilerInstance&, int&, swift::FrontendObserver*) + 350
24 swift-frontend           0x000000010d67bf5e swift::performFrontend(llvm::ArrayRef<char const*>, char const*, void*, swift::FrontendObserver*) + 5214
25 swift-frontend           0x000000010d610e42 main + 866
26 libdyld.dylib            0x00007fff203e4621 start + 1
error: Abort trap: 6 (in target 'MediaBeam' from project 'MediaBeam')

+0.9 - Async sequences would be a very useful addition to Swift. My only concerns are related to areas where the design of this feature diverges from other similar features in mainstream languages.

cancel isn't async

I'm familiar with both .Net await foreach / IAsyncEnumerable and JavaScript's for-await-of / Symbol.asyncIterator and this proposal seems to follow the same structure with one exception: the design of func cancel(). The .Net equivalent is DisposeAsync() and the JS equivalent is return(). These return (Value)Task/Promise respectively, meaning that they are asynchronous operations. I'm not saying the design in Swift needs to be identical, but I'd like to understand why you think Swift is unlikely to need asynchronous cleanup for early exit of a for await?

An example use case would be if the AsyncSequence represents the query results from a database, you might need network access to close the cursor, which would imply making it async.

If/when Swift supports generators for implementing (Async)IteratorProtocol, this would correspond to supporting await within defer blocks in the generator. I note that this isn't currently supported, so a synchronous func cancel() is at least consistent with that.

How to cancel an iteration from outside the loop

I also had questions about how this interacts with cancellation of the iteration from outside the loop / AsyncSequence functions. For example, if a user interface triggers an operation that maps over an AsyncSequence backed by a paged web service API. How could this operation be cancelled? I think this would be supported by something like

struct API {
  func query() -> AsyncSequence<APIResult>
}
let handle = Task.runDetached { self.results = api.query().map { ... } }
...
handle.cancel()

Is this correct?

Naming of func cancel()

Would this be better named something else? I found the name of this confusing when I was thinking about the above case of cancelling from outside the loop. It seems like this is actually doing some cleanup after cancellation. .Net has Dispose(Async) to standardise this. JS calls this return. If it's not going to be part of deinit, would a name like close, finalise, cleanup be clearer?

Yes and yes.

I've used asyncIterator + IxJS in JavaScript for processing CSV files that were too big to fit in memory and they made the computation much more comprehensible than they would otherwise have been using Node streams.

Took part in the Pitch thread, read the proposal several times and compared the structure to similar features in .Net + JavaScript.

3 Likes

Thanks for this comment, it's a good question.

We imagined cancel as best-effort, and cooperative. I think many (most?) cases would not require an async cancel function; probably they would just set some flag which the actual async work could query. Making it async would require all exits from the for loop to be marked as await, which would complicate the call site quite a bit. (n.b. even if we did away with cancel completely, that would just move this same problem to deinit, which afaik does not allow async in the first place).

For this database case, would it be reasonable to have this particular iterator run a detached task to close the cursor? What would happen in case the network was unreachable?

The example as written doesn't actually do anything asynchronously. However if you add a for loop over the results of map, that would be asynchronous and cancelling the task must propagate into the loop as well. I will double check on our prototype implementation.

The participation in the structured concurrency proposal for cancellation is a big reason why I think using the name cancel here actually makes a lot of sense.

1 Like

I would argue the opposite and argue that cancel() is very important. The reason that cancel is important is that deinit only works if you can guarantee all of the following conditions are met for all resources:

  • tear down can be done from any thread (we can't guarantee the thread deinit is called on)
  • tear down can be done synchronously (we can't keep self alive after deinit, so blocking is the only way out)
  • tear down can be done from almost arbitrary contexts (we can't guarantee where the deinit call will come from)

Very often, not all of these are true. Especially for kernel resources we run into very deep issues really fast. [Don't get me wrong: deinit works wonderfully for memory because releasing memory is fast, can be done on any thread, and in any context (that a Swift program can be in)]. In fact, some good old docs have some key information too (still in ObjC...): "Don't Use dealloc to Manage Scarce Resources" (like file descriptors).

As an example, let's take an AsyncIterator that delivers the bytes of a network connection. We will probably have to close the network connection before we discard our async iterator (or else we'd leak file descriptors). Before we can however close the file descriptor, we'll have to unregister it from the eventing system (DispatchSource, kqueue, epoll, ...) and unregistration is usually not a synchronous operation. So the only way I could see how one would use deinit is to start the deregistration, block until it completes (this may be never because it may need the current thread to complete the deregistration), then close the file descriptor.

Why does the same issue not also apply to an explicit cancel()? At first, cancel() looks kind of similar to deinit, both are synchronous "methods". But there is a crucial difference: In cancel() I can keep the current object around for as long as I want. For example when a class instance is responsible for a kernel resource, it's common to create a deliberate retain cycle which is broken when the tear down of the resource is complete. So after a call to cancel we would initiate the tear down, often driven by a state machine that understands what tear down operations have to happen in what order. We can keep the instance (with all the associated state (machines)) alive until we're done with the resource, break the retain cycle, and we can be sure to not leak resources.

And note that the proposed "cancel is not called if the iterator ended" is totally fine. The iterator itself knows when it returned nil which is a fine point to also initiate the tear down. cancel would just be a way to initiate teardown before the sequence has finished.

8 Likes

I'm afraid I don't have a good answer for that. Would subsequent code rely on the cleanup being complete?

I did find the following discussion on the design of DisposeAsync() plus other background for allowing async cleanup in general that might be useful.

Sorry - I should have been clearer. I think the idea I had was that something in var results { set } would be doing the iteration.

I don't have a particularly strong objection to the current name, just found it a little confusing.

I agree with both @johannesweiss and @benlings here.

  1. We should prefer explicit cancel over relying on deinit for all the reasons @johannesweiss spelled out. I consider doing cleanup of asynchronous work in deinit to be a code smell because it's rarely done in a safe way. If you need to stop an asynchronous task then you should have an explicit method to do so.
  2. Stopping asynchronous work can in some cases be an asynchronous task in itself, which means cancel should be able to be async.

This same issue came up when async/await itself was under review in response to the suggestion to allow await inside a defer block. I still maintain that it's ok to have an implicit suspension when exiting a scope due to an await in a defer block because the actual suspension is made visible at the site of that await in the defer block.

Here is slightly different because the only await is the for/in loop itself. Obviously the C# designers considered the same issue (as seen by the document linked by @benlings), but they were willing to live with the implicit await. Personally I agree with them that it's better to allow for the flexibility.

That said, I also wonder about the possibility of using @asyncHandler inside an implementation of cancel. If we don't actually need to wait for the cancel to finish then maybe it's ok?

I understand what you're saying here, and have seen several of those issues in GC'd systems, but Swift is not Java ;-)

At least in the case of for-in loop, cancel and deinit are equally expressive, none of these points apply, because the iterator is guaranteed to be destroyed on exit from the loop by ARC.

There is a point here about direct use of the AsyncIterator API which is valid, but I still don't think a lot of those points translate over to ARC based systems with memory isolation like Swift-with-concurrency will be.

-Chris

All GC'd systems I've used don't have guaranteed finalisation and also even adding a finaliser usually "makes things slow" so I've never really seen them used. ARC is quite different because it guarantees that deinit is called (if there are no cycles), however a programmer still can't reliably know when/where exactly deinit is called without understanding every intricacy of every single line of code.

I must be misunderstanding something then.

Let's take this piece of code

func stupidLog(_ message: @autoclosure () -> String) {
    DispatchQueue.main.async {
        print("log: \(message())")
    }
}

let iterator = SomeAsyncIterator(...)
for x in iterator {
    print(x)
    break
}
stupidLog("done with \(iterator)")

If I'm not misunderstanding, the generated code will look something like

if let x = iterator.next() {
    print(x)
    iterator.cancel() // because of the break [this line would be missing without `cancel` ofc]
}
iterator.retain() // to capture `iterator`
stupidLog(Closure(closureFunctionPointer, closureContext /* containing iterator */))
iterator.release() // to drop one reference to iterator --> likely NO DEINIT because the closure has one ref

So whether deinit is called at the end of the above code depends on when/if the main queue will run our "stupidLog" closure. With cancel, there is no question and it will run for sure right when the loop exits. We cannot guarantee the same with ARC because we can always escape a reference to the iterator to elsewhere.

But regardless if deinit is run here or not, my main argument is that we cannot [reasonably easily] tear down any kind of resource in deinit because many resources cannot be torn down synchronously. I understand that in theory, we could create a new object in deinit, transfer the ownership of the resource to be torn down over to the new object and let the new object deal with the tear down. But that seems unreasonably hard and error-prone.

That's my point: the code generation for the for-in loop (not direct use of the API) completely encapsulates the iterator. It is exactly as expressive as deinit. That was the only point I was trying to make. I agree that manual use of the API is a different thing.

Right, you are understanding ARC correctly, one nuance is that there cannot be an implicit escape across threads, because the iterator doesn't conform to the right ActorSendable/ConcurrentValue protocol.

The issue you're observing (resource cleanup with ARC requires knowing where deinit runs) is pervasive in Swift (and I expect to get somewhat better with ownership), it isn't specific to AsyncSequence. By your argument, we should add cancel to all non-trivial types because destruction isn't predictable enough.

-Chris

3 Likes

Is it guaranteed? I know Objective-C isn't Swift either, but the issues I described were seen in Objective-C with ARC. If anything ARC makes it harder to reason about when/where objects are deallocated (relative to MRR) because of implicit captures/retains/releases. As long as the enumerator is a refcounted object this seems like a questionable guarantee.

I don't think that's a fair argument. We're specifically talking about an object that represents asynchronous work. I would definitely argue that anything that represents asynchronous work should either not be cancellable at all or should be explicitly cancellable (not via deinit). Most types don't represent asynchronous work, though.

I did not take any potential future work of ActorSendable/ConcurrentValue into account. This will improve things but just a bit. The same argument could be made with somebody holding onto iterator for a little longer, we just don't know (without move-only types) when/if deinit is called. File descriptors and other resources are scarce so prompt & very reliable termination is much more crucial than with memory.

I agree with all of this. It's indeed not specific to AsyncSequence but many (most?) use cases for AsyncSequence may hold onto system resources (like network sockets, file descriptors, ...) which usually can't be torn down synchronously.

I'm not sure I follow why it should be added to all non-trivial types. Adding it to IteratorProtocol would make sense to me but I've just not seen the synchronous IteratorProtocol used for networking/reading files etc because those things are typically done in an asynchronous way. I think it is just much more likely that something asynchronous has attached system resources over something synchronous.

But also you're leaving out the core of my argument: It's very very hard to even implement asynchronous resource teardown in deinit. Even if we had move-only types and/or perfect knowledge of where deinit runs.

FWIW, in the server ecosystem we do usually require people to do manual resource termination if something is attached to a scarce resource (files/sockets/...). But we do (in debug mode usually) use deinit to validate that in fact the user has called appropriate destruction method (example here).

Yes, ObjC and Swift ARC are often completely different, e.g. ObjC has all sorts of interesting things with autorelease pools that have nothing to do with Swift (outside of bridging). Also, here we are talking about a language feature, not general API use.

The iterator for a for-in loop is guaranteed to be destroyed on exit. This is one of the reasons that cancel is defined as __consuming to make sure this aligns with future ownership things.

I agree with this -- this argues that cancel should be marked async if present. Tony makes very good points upthread about why that is a bad idea, which is further evidence for "we shouldn't support this" in my opinion.

-Chris

But this is a protocol in the standard library, right? So should there also be a compiler feature to prevent direct usages? Are there other such API features in the standard library that shouldn't be used directly?

It seems like either the API should be generally usable and useful (directly) or it shouldn't be possible to use in any other way. Otherwise it will be used in a way that was not intended...

It's substantially easier (we do this all the time) to implement asynchronous resource teardown with a synchronous cancel than it is with deinit. The reason is that in cancel I don't lose access to my state machine at the end of the cancel method, with deinit I do.

Example (it ignores threading and doesn't do the actual reading of bytes, just demonstrates setting up/tearing down the eventing and the file descriptor as a system resource):

import Dispatch

class ReadFDIterator: AsyncIteratorProtocol {
    typealias Element = [UInt8]

    private enum State {
        case activated
        case cancellingEventing
        case resourcesDropped
    }

    private let fd: CInt
    private let readSource: DispatchSourceRead
    private var state = State.activated

    private func cancelHandler() {
        // When this is called, we regain ownership of the fd and can close it.
        // Closing it before is a use-after free.
        assert(self.state == .cancellingEventing)
        self.state = .resourcesDropped

        close(self.fd)
    }

    init(fileDescriptor: CInt) {
        self.fd = fileDescriptor
        // From this point on, Dispatch owns the file descriptor, we only get it back once the
        // cancel handler has been called. We cannot close the fd before.
        self.readSource = DispatchSource.makeReadSource(fileDescriptor: fileDescriptor)

        // Deliberate retain cycle to keep `self` alive until `self.cancelHandler` has been called.
        self.readSource.setCancelHandler(handler: self.cancelHandler)

        // Kick off the eventing.
        self.readSource.activate()
    }

    func next() async throws -> [UInt8]? {
        let element = ... // do the actual work
        if element == nil {
            self.cancel() // no need to duplicate the logic in this case
        }
        return element
    }

    func cancel() {
        switch self.state {
        case .activated:
            self.state = .cancellingEventing
            self.readSource.cancel() // request cancellation
        case .cancellingEventing, .resourcesDropped:
            () // nothing to do
        }
    }

    deinit {
        assert(self.state == .resourcesDropped)
    }
}

See how it's pretty straightforward to implement cancel? We just request cancellation and actually tear down the resource when Dispatch calls the cancelHandler. This would not be possible with deinit because we're dead after we return.

4 Likes

I must be missing something. I don't see how it's different from:

init(fileDescriptor: CInt) {
  ...
  self.readSource.setCancelHandler {
    close(fileDescriptor)
  }
}

deinit {
  readSource.cancel() // Do we need this?
}
Terms of Service

Privacy Policy

Cookie Policy