Pitch: Progress Tracking in Swift


(Charles Srstka) #1

Introduction:

This is a proposal for a native progress-tracking mechanism in Swift.

Motivation:

As most of us know, Foundation includes NSProgress, a class that makes it fairly simple to implement progress tracking in complex applications. What makes NSProgress nice is the way that it builds a tree of progress objects, so that each method or function only needs to take into account the work that needs to be done in that specific context, and the parent NSProgress object can interpret that progress in the context of its own work to be done. NSProgress also provides a support for common issues like checking whether the user has cancelled an operation. However, there are a few issues with NSProgress:

1. Because NSProgress relies on a thread-local global variable to store the current progress object, it is impossible to know, outside of documentation, whether any particular method supports NSProgress or not, and if the documentation is inadequate, one must resort to trial-and-error to determine whether NSProgress is supported (this has been noted before: http://oleb.net/blog/2014/03/nsprogress/).

2. NSProgress relies on a paired register-deregister pattern in order to obtain and release "current" status. One must balance every becomeCurrentWithPendingUnitCount() call with a resignCurrent() call, or Bad Things™ will happen. However, this pattern does not work well with Swift's error handling mechanism:

progress.becomeCurrentWithPendingUnitCount(1)
try self.doSomeThingThatMightThrow()
progress.resignCurrent() // If an error occurs, this will never be called!

3. It is very easy to accidentally add a child NSProgress object if one does not realize that a particular method supports NSProgress. A couple ways this can happen:

func doSomething() {
     let progress = NSProgress(totalUnitCount: 10)

     progress.becomeCurrentWithPendingUnitCount(10)
     doSomethingThatSupportsNSProgress()
     doSomethingThatWeDontExpectSupportsNSProgressButDoes() // whoops, we just picked up a child accidentally
     progress.resignCurrent()
}

func doSomething() {
     let progress = NSProgress(totalUnitCount: 10)

     progress.becomeCurrentWithPendingUnitCount(10)
     doSomethingThatSupportsNSProgress()
     progress.resignCurrent()

     doSomethingThatWeDontExpectSupportsNSProgressButDoes() // whoops, this one just picked up our parent's NSProgress and made it our sibling
}

This is particularly problematic when you consider the obvious workaround for the problem described in #2:

func doSomething() {
     let progress = NSProgress(totalUnitCount: 10)

     progress.becomeCurrentWithPendingUnitCount(10)
     defer { progress.resignCurrent() }

     doSomethingThatSupportsNSProgress()
     doSomethingThatWeDontExpectSupportsNSProgressButDoes() // whoops
}

I haven't figured out exactly how to reproduce this reliably yet, but I have noticed that NSProgress objects can sometimes end up with a fractionCompleted larger than 1.0 when this occurs.

4. Because NSProgress uses a thread-local global variable for the current progress, one must be vigilant about jumping through specific hoops when spinning off a task in a new thread, or else the child task's NSProgress object will not be properly connected to its parent.

5. NSProgress posts KVO notifications on the main thread. In addition to complicating the interface and causing undefined behavior if one mistakenly binds it to a UI element without inserting something in between to forward the KVO notifications to the main thread, this also hurts performance in the worker thread, since KVO notifications carry a non-trivial performance penalty, involving not only message sends but also packaging things into an NSDictionary for the "change" parameter. Notifications for multiple properties are fired every time completedUnitCount is changed, which then can cause similar KVO notifications to occur up the family tree, depending on how many KVO observers are installed. This can show up as a non-trivial performance cost in Instruments if the NSProgress object is updated frequently enough.

Proposed Solution:

I propose a system that follows the example of Swift's excellent do/try/catch error-handling system. A "reports" keyword can be added to a function or method's declaration, similar to "throws":

func myFunc() reports -> () {
     ...
}

By default, the "reports" keyword will introduce a "let" constant named "progress" to the function's scope, much like "catch" implicitly adds "error". The name of this constant could be customized, if desired:

func myFunc() reports(myNiftyProgressObject) {
     ...
}

The created progress object is automatically added as a child to the progress object which is implicitly passed into the function by the calling function or method.

The interface to the progress object would be similar to NSProgress, with a "prepare" method taking the total unit count, as well as a "cancelled" property and a "cancel()" function helping keep track of whether or not an operation is cancelled. However, the "becomeCurrent" and "resignCurrent" methods are replaced with a "report" keyword which precedes a function or method call, similar to "try", although the "report" keyword includes a parameter taking the number of units corresponding to this operation:

func myFunc() throws, reports {
     progress.prepare(10)

     report(5) foo()

     if progress.cancelled { throw NSCocoaError.UserCancelledError }

     report(5) try bar() // yes, "report" can be combined with "try"
}

Spinning off an operation onto another thread or queue is no problem:

func myFunc() reports {
     progress.prepare(10)

     dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0)) {
         report(10) foo() // the progress object is passed as a parameter to the function, so no need to specially make it current on this thread first
     }
}

Unlike "try", "report" is not required. If we don't care about the progress of a method or function, we can call it without "report" and its progress will be ignored. In this example, the "someRelativelyTrivialOperationThatReports"'s progress object will *not* be added as a child to "progress":

func myFunc() reports {
     progress.prepare(10)

     someRelativelyTrivialOperationThatReports()
     report(10) someNonTrivialOperationThatReports()
}

If a function calls only one reporting function, the "prepare" method can be omitted and the parameter can be left off of "report", and the parent progress object will simply be passed through:

func myFunc() reports {
     report doAllTheActualWork()
}

To pass in a progress object other than the default progress object (for example, when creating the object at the very top of the tree), simply pass the progress object in as an optional second parameter to "reports":

func startEverything() {
     let progress = Progress()

     progress.prepare(10)

     report(10, progress) someFuncThatReports()
}

In the function that does the actual work, of course the progress object has a "completed" property that tracks the completed units:

func doTheActualWork() reports {
     let someArrayOfThings = ...

     progress.prepare(someArrayOfThings.count)

     for eachThing in someArrayOfThings {
         // do the work

         progress.completed += 1
     }
}

Observing the progress object uses a closure-based system, containing a "willUpdateHandler", a "didUpdateHandler", and a "cancelHandler":

progress.didUpdateHandler = { (completed: Int, total: Int) -> () in
     print("\(completed) units complete out of \(total)")
}

progress.cancelHandler = {
     print("the user cancelled the operation")
}

The separation between will- and did- handlers is to facilitate interoperability with NSProgressIndicator and other Objective-C objects; one could call willChangeValueForKey() and didChangeValueForKey() in the handlers to let the KVO system know about the changes if needed. If desired, a small wrapper could be added to Foundation to translate the progress object's notifications into KVO notifications.

The progress object also includes a property specifying the dispatch queue on which the handlers will be run. By default this is the main queue, but this can be customized as needed:

progress.dispatchQueue = myDispatchQueue

Of course, any changes to the "completed" property in a child progress object would be bubbled up the family tree, causing any handlers on the parent progress objects to fire, and any changes to the "cancelled" property in a parent progress object would be propagated to its children, similar to how NSProgress currently works.

If there is interest in this, I could flesh out the interface and implementation to the progress object a bit.

What do you all think?

Charles


(Charles Srstka) #2

Typo here: This should have said "on the current thread." The fact that NSProgress bogs down the worker thread with the work involved in creating the KVO notifications is part of my issue with it.

Charles

···

On 2016-01-17 21:14, Charles Srstka via swift-evolution wrote:

5. NSProgress posts KVO notifications on the main thread.


(Rob Mayoff) #3

1. Because NSProgress relies on a thread-local global variable to store
the current progress object, it is impossible to know, outside of
documentation, whether any particular method supports NSProgress or not,
and if the documentation is inadequate, one must resort to trial-and-error
to determine whether NSProgress is supported (this has been noted before:
http://oleb.net/blog/2014/03/nsprogress/).

iOS 9 and OS X 10.11 support explicit progress composition through the
addChild:withPendingUnitCount: message, and vending of an NSProgress
instance (for use with the addChild:withPendingUnitCount: message) through
the NSProgressReporting protocol. You no longer have to rely on implicit
composition, and in WWDC 2015 session 232 you're encouraged to use explicit
composition.

I have a hard time believing this is something that needs dedicated
language syntax.


(Brent Royal-Gordon) #4

What do you all think?

I think this is really thin syntactic sugar over `-> NSProgress`, and doesn't add enough value to justify itself. Unlike the error handling system:

- The fact that a function *can* provide progress doesn't mean that any particular caller needs that progress.
- Progress monitoring doesn't require a lot of convoluted control flow.
- There is no need to customize the progress monitoring system in the way that, for instance, `ErrorType` allows.
- Progress monitoring doesn't require convoluted ways of returning the progress, since most progress-reporting operations are asynchronous and have no other immediate return value.
- Progress monitoring is actually done fairly infrequently.

I think that all Swift needs to support progress monitoring is to make sure that `NSProgress` returns can be ignored without a warning even if we make `@warn_unused_return` the default, and perhaps to overload `+=` on two `NSProgress`es if we can find an appropriate way to do it.

···

--
Brent Royal-Gordon
Architechies


(Tony Parker) #5

Hi Charles,

I see Rob already pointed this out, but I’ll respond as well since I own NSProgress and I also was involved in the WWDC session he references.

NSProgress actually received a significant upgrade in iOS 9 and OS X 10.11 to address most of your concerns. I’ve never really liked the ‘current’ progress idea either, so I’ve changed Foundation to make using explicit progress objects possible (and encouraged). There are still cases where the idea of having a current progress is useful. For example, when there is no opportunity to grab a progress object, because the method you are calling is a proxy, or a global function, etc.).

Introduction:

This is a proposal for a native progress-tracking mechanism in Swift.

Motivation:

As most of us know, Foundation includes NSProgress, a class that makes it fairly simple to implement progress tracking in complex applications. What makes NSProgress nice is the way that it builds a tree of progress objects, so that each method or function only needs to take into account the work that needs to be done in that specific context, and the parent NSProgress object can interpret that progress in the context of its own work to be done. NSProgress also provides a support for common issues like checking whether the user has cancelled an operation. However, there are a few issues with NSProgress:

1. Because NSProgress relies on a thread-local global variable to store the current progress object, it is impossible to know, outside of documentation, whether any particular method supports NSProgress or not, and if the documentation is inadequate, one must resort to trial-and-error to determine whether NSProgress is supported (this has been noted before: http://oleb.net/blog/2014/03/nsprogress/).

You can now create progress trees explicitly. There are two parts to this:

    @available(OSX 10.11, *)
    public /*not inherited*/ init(totalUnitCount unitCount: Int64, parent: NSProgress, pendingUnitCount portionOfParentTotalUnitCount: Int64)

and a new protocol:

public protocol NSProgressReporting : NSObjectProtocol {
    @available(OSX 10.9, *)
    public var progress: NSProgress { get }
}

So you can mark classes that publish progress with the protocol, and clients can create their trees using the new initializer.

2. NSProgress relies on a paired register-deregister pattern in order to obtain and release "current" status. One must balance every becomeCurrentWithPendingUnitCount() call with a resignCurrent() call, or Bad Things™ will happen. However, this pattern does not work well with Swift's error handling mechanism:

progress.becomeCurrentWithPendingUnitCount(1)
try self.doSomeThingThatMightThrow()
progress.resignCurrent() // If an error occurs, this will never be called!

Yes, agreed. I would encourage the discrete progress if possible, but I’ve had lots of requests for a block-based version of becomeCurrent. This is another good reason to add one.

3. It is very easy to accidentally add a child NSProgress object if one does not realize that a particular method supports NSProgress. A couple ways this can happen:

func doSomething() {
   let progress = NSProgress(totalUnitCount: 10)

   progress.becomeCurrentWithPendingUnitCount(10)
   doSomethingThatSupportsNSProgress()
   doSomethingThatWeDontExpectSupportsNSProgressButDoes() // whoops, we just picked up a child accidentally
   progress.resignCurrent()
}

func doSomething() {
   let progress = NSProgress(totalUnitCount: 10)

   progress.becomeCurrentWithPendingUnitCount(10)
   doSomethingThatSupportsNSProgress()
   progress.resignCurrent()

   doSomethingThatWeDontExpectSupportsNSProgressButDoes() // whoops, this one just picked up our parent's NSProgress and made it our sibling
}

This is particularly problematic when you consider the obvious workaround for the problem described in #2:

func doSomething() {
   let progress = NSProgress(totalUnitCount: 10)

   progress.becomeCurrentWithPendingUnitCount(10)
   defer { progress.resignCurrent() }

   doSomethingThatSupportsNSProgress()
   doSomethingThatWeDontExpectSupportsNSProgressButDoes() // whoops
}

I haven't figured out exactly how to reproduce this reliably yet, but I have noticed that NSProgress objects can sometimes end up with a fractionCompleted larger than 1.0 when this occurs.

I changed the algorithm for how we add child progress objects, so now we only will add the first one. This wasn’t strictly backwards compatible, but it was far too easy to add more progress objects than you intended.

4. Because NSProgress uses a thread-local global variable for the current progress, one must be vigilant about jumping through specific hoops when spinning off a task in a new thread, or else the child task's NSProgress object will not be properly connected to its parent.

Yup, again another problem solved by using the discrete progress objects.

5. NSProgress posts KVO notifications on the main thread.

This actually isn’t true. NSProgress posts KVO notifications on the thread in which the change was made.

In addition to complicating the interface and causing undefined behavior if one mistakenly binds it to a UI element without inserting something in between to forward the KVO notifications to the main thread, this also hurts performance in the worker thread, since KVO notifications carry a non-trivial performance penalty, involving not only message sends but also packaging things into an NSDictionary for the "change" parameter. Notifications for multiple properties are fired every time completedUnitCount is changed, which then can cause similar KVO notifications to occur up the family tree, depending on how many KVO observers are installed. This can show up as a non-trivial performance cost in Instruments if the NSProgress object is updated frequently enough.

There is a cost associated with updating progress, it’s true. We can always do more to optimize it (and we continue to do so), but there comes a point at which the updater of the progress object has to add some intelligence of its own. This is why the documentation and release notes encourage callers to avoid doing things like updating the progress for each byte in a file (because that is far too many updates).

- Tony

···

On Jan 17, 2016, at 6:14 PM, Charles Srstka via swift-evolution <swift-evolution@swift.org> wrote:

Proposed Solution:

I propose a system that follows the example of Swift's excellent do/try/catch error-handling system. A "reports" keyword can be added to a function or method's declaration, similar to "throws":

func myFunc() reports -> () {
   ...
}

By default, the "reports" keyword will introduce a "let" constant named "progress" to the function's scope, much like "catch" implicitly adds "error". The name of this constant could be customized, if desired:

func myFunc() reports(myNiftyProgressObject) {
   ...
}

The created progress object is automatically added as a child to the progress object which is implicitly passed into the function by the calling function or method.

The interface to the progress object would be similar to NSProgress, with a "prepare" method taking the total unit count, as well as a "cancelled" property and a "cancel()" function helping keep track of whether or not an operation is cancelled. However, the "becomeCurrent" and "resignCurrent" methods are replaced with a "report" keyword which precedes a function or method call, similar to "try", although the "report" keyword includes a parameter taking the number of units corresponding to this operation:

func myFunc() throws, reports {
   progress.prepare(10)

   report(5) foo()

   if progress.cancelled { throw NSCocoaError.UserCancelledError }

   report(5) try bar() // yes, "report" can be combined with "try"
}

Spinning off an operation onto another thread or queue is no problem:

func myFunc() reports {
   progress.prepare(10)

   dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0)) {
       report(10) foo() // the progress object is passed as a parameter to the function, so no need to specially make it current on this thread first
   }
}

Unlike "try", "report" is not required. If we don't care about the progress of a method or function, we can call it without "report" and its progress will be ignored. In this example, the "someRelativelyTrivialOperationThatReports"'s progress object will *not* be added as a child to "progress":

func myFunc() reports {
   progress.prepare(10)

   someRelativelyTrivialOperationThatReports()
   report(10) someNonTrivialOperationThatReports()
}

If a function calls only one reporting function, the "prepare" method can be omitted and the parameter can be left off of "report", and the parent progress object will simply be passed through:

func myFunc() reports {
   report doAllTheActualWork()
}

To pass in a progress object other than the default progress object (for example, when creating the object at the very top of the tree), simply pass the progress object in as an optional second parameter to "reports":

func startEverything() {
   let progress = Progress()

   progress.prepare(10)

   report(10, progress) someFuncThatReports()
}

In the function that does the actual work, of course the progress object has a "completed" property that tracks the completed units:

func doTheActualWork() reports {
   let someArrayOfThings = ...

   progress.prepare(someArrayOfThings.count)

   for eachThing in someArrayOfThings {
       // do the work

       progress.completed += 1
   }
}

Observing the progress object uses a closure-based system, containing a "willUpdateHandler", a "didUpdateHandler", and a "cancelHandler":

progress.didUpdateHandler = { (completed: Int, total: Int) -> () in
   print("\(completed) units complete out of \(total)")
}

progress.cancelHandler = {
   print("the user cancelled the operation")
}

The separation between will- and did- handlers is to facilitate interoperability with NSProgressIndicator and other Objective-C objects; one could call willChangeValueForKey() and didChangeValueForKey() in the handlers to let the KVO system know about the changes if needed. If desired, a small wrapper could be added to Foundation to translate the progress object's notifications into KVO notifications.

The progress object also includes a property specifying the dispatch queue on which the handlers will be run. By default this is the main queue, but this can be customized as needed:

progress.dispatchQueue = myDispatchQueue

Of course, any changes to the "completed" property in a child progress object would be bubbled up the family tree, causing any handlers on the parent progress objects to fire, and any changes to the "cancelled" property in a parent progress object would be propagated to its children, similar to how NSProgress currently works.

If there is interest in this, I could flesh out the interface and implementation to the progress object a bit.

What do you all think?

Charles

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Dave Abrahams) #6

I think this is really out-of-scope for swift-evolution. There's little
chance we'd want something like this in the standard library in the near
term. It's really Foundation's domain.

-Dave

···

on Sun Jan 17 2016, Charles Srstka via swift-evolution <swift-evolution-m3FHrko0VLzYtjvyW6yDsg-AT-public.gmane.org> wrote:

Introduction:

This is a proposal for a native progress-tracking mechanism in Swift.

Motivation:

As most of us know, Foundation includes NSProgress, a class that makes
it fairly simple to implement progress tracking in complex
applications. What makes NSProgress nice is the way that it builds a
tree of progress objects, so that each method or function only needs
to take into account the work that needs to be done in that specific
context, and the parent NSProgress object can interpret that progress
in the context of its own work to be done. NSProgress also provides a
support for common issues like checking whether the user has cancelled
an operation. However, there are a few issues with NSProgress:

1. Because NSProgress relies on a thread-local global variable to
store the current progress object, it is impossible to know, outside
of documentation, whether any particular method supports NSProgress or
not, and if the documentation is inadequate, one must resort to
trial-and-error to determine whether NSProgress is supported (this has
been noted before: http://oleb.net/blog/2014/03/nsprogress/).

2. NSProgress relies on a paired register-deregister pattern in order
to obtain and release "current" status. One must balance every
becomeCurrentWithPendingUnitCount() call with a resignCurrent() call,
or Bad Things™ will happen. However, this pattern does not work well
with Swift's error handling mechanism:

progress.becomeCurrentWithPendingUnitCount(1)
try self.doSomeThingThatMightThrow()
progress.resignCurrent() // If an error occurs, this will never be
called!

3. It is very easy to accidentally add a child NSProgress object if
one does not realize that a particular method supports NSProgress. A
couple ways this can happen:

func doSomething() {
    let progress = NSProgress(totalUnitCount: 10)

    progress.becomeCurrentWithPendingUnitCount(10)
    doSomethingThatSupportsNSProgress()
    doSomethingThatWeDontExpectSupportsNSProgressButDoes() // whoops,
we just picked up a child accidentally
    progress.resignCurrent()
}

func doSomething() {
    let progress = NSProgress(totalUnitCount: 10)

    progress.becomeCurrentWithPendingUnitCount(10)
    doSomethingThatSupportsNSProgress()
    progress.resignCurrent()

    doSomethingThatWeDontExpectSupportsNSProgressButDoes() // whoops,
this one just picked up our parent's NSProgress and made it our
sibling
}

This is particularly problematic when you consider the obvious
workaround for the problem described in #2:

func doSomething() {
    let progress = NSProgress(totalUnitCount: 10)

    progress.becomeCurrentWithPendingUnitCount(10)
    defer { progress.resignCurrent() }

    doSomethingThatSupportsNSProgress()
    doSomethingThatWeDontExpectSupportsNSProgressButDoes() // whoops
}

I haven't figured out exactly how to reproduce this reliably yet, but
I have noticed that NSProgress objects can sometimes end up with a
fractionCompleted larger than 1.0 when this occurs.

4. Because NSProgress uses a thread-local global variable for the
current progress, one must be vigilant about jumping through specific
hoops when spinning off a task in a new thread, or else the child
task's NSProgress object will not be properly connected to its parent.

5. NSProgress posts KVO notifications on the main thread. In addition
to complicating the interface and causing undefined behavior if one
mistakenly binds it to a UI element without inserting something in
between to forward the KVO notifications to the main thread, this also
hurts performance in the worker thread, since KVO notifications carry
a non-trivial performance penalty, involving not only message sends
but also packaging things into an NSDictionary for the "change"
parameter. Notifications for multiple properties are fired every time
completedUnitCount is changed, which then can cause similar KVO
notifications to occur up the family tree, depending on how many KVO
observers are installed. This can show up as a non-trivial performance
cost in Instruments if the NSProgress object is updated frequently
enough.

Proposed Solution:

I propose a system that follows the example of Swift's excellent
do/try/catch error-handling system. A "reports" keyword can be added
to a function or method's declaration, similar to "throws":

func myFunc() reports -> () {
    ...
}

By default, the "reports" keyword will introduce a "let" constant
named "progress" to the function's scope, much like "catch" implicitly
adds "error". The name of this constant could be customized, if
desired:

func myFunc() reports(myNiftyProgressObject) {
    ...
}

The created progress object is automatically added as a child to the
progress object which is implicitly passed into the function by the
calling function or method.

The interface to the progress object would be similar to NSProgress,
with a "prepare" method taking the total unit count, as well as a
"cancelled" property and a "cancel()" function helping keep track of
whether or not an operation is cancelled. However, the "becomeCurrent"
and "resignCurrent" methods are replaced with a "report" keyword which
precedes a function or method call, similar to "try", although the
"report" keyword includes a parameter taking the number of units
corresponding to this operation:

func myFunc() throws, reports {
    progress.prepare(10)

    report(5) foo()

    if progress.cancelled { throw NSCocoaError.UserCancelledError }

    report(5) try bar() // yes, "report" can be combined with "try"
}

Spinning off an operation onto another thread or queue is no problem:

func myFunc() reports {
    progress.prepare(10)

    dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT,
0)) {
        report(10) foo() // the progress object is passed as a
parameter to the function, so no need to specially make it current on
this thread first
    }
}

Unlike "try", "report" is not required. If we don't care about the
progress of a method or function, we can call it without "report" and
its progress will be ignored. In this example, the
"someRelativelyTrivialOperationThatReports"'s progress object will
*not* be added as a child to "progress":

func myFunc() reports {
    progress.prepare(10)

    someRelativelyTrivialOperationThatReports()
    report(10) someNonTrivialOperationThatReports()
}

If a function calls only one reporting function, the "prepare" method
can be omitted and the parameter can be left off of "report", and the
parent progress object will simply be passed through:

func myFunc() reports {
    report doAllTheActualWork()
}

To pass in a progress object other than the default progress object
(for example, when creating the object at the very top of the tree),
simply pass the progress object in as an optional second parameter to
"reports":

func startEverything() {
    let progress = Progress()

    progress.prepare(10)

    report(10, progress) someFuncThatReports()
}

In the function that does the actual work, of course the progress
object has a "completed" property that tracks the completed units:

func doTheActualWork() reports {
    let someArrayOfThings = ...

    progress.prepare(someArrayOfThings.count)

    for eachThing in someArrayOfThings {
        // do the work

        progress.completed += 1
    }
}

Observing the progress object uses a closure-based system, containing
a "willUpdateHandler", a "didUpdateHandler", and a "cancelHandler":

progress.didUpdateHandler = { (completed: Int, total: Int) -> () in
    print("\(completed) units complete out of \(total)")
}

progress.cancelHandler = {
    print("the user cancelled the operation")
}

The separation between will- and did- handlers is to facilitate
interoperability with NSProgressIndicator and other Objective-C
objects; one could call willChangeValueForKey() and
didChangeValueForKey() in the handlers to let the KVO system know
about the changes if needed. If desired, a small wrapper could be
added to Foundation to translate the progress object's notifications
into KVO notifications.

The progress object also includes a property specifying the dispatch
queue on which the handlers will be run. By default this is the main
queue, but this can be customized as needed:

progress.dispatchQueue = myDispatchQueue

Of course, any changes to the "completed" property in a child progress
object would be bubbled up the family tree, causing any handlers on
the parent progress objects to fire, and any changes to the
"cancelled" property in a parent progress object would be propagated
to its children, similar to how NSProgress currently works.

If there is interest in this, I could flesh out the interface and
implementation to the progress object a bit.

What do you all think?


(Will Entriken) #7

Charles,

I think this is interesting. And discussion on replacing/updating
NSProgress is important.

Just a note: optional parameters make this possible without changing the
language. A very rough example follows:

func delete(files: FileList, withProgress progress: Progress? = nil) {
    progress.totalUnits = 10
    print("Getting started")
    progress.completedUnits = 5 // that was easy!
    actuallyDelete(files, withProgress: progress.childWithUnits: 5)
}

let progress = Progress()
// watch progress.totalPortion using KVO 2.0
delete(files: allTheFiles, withProgress: progress)

Regards,
Will

···

On Sun, Jan 17, 2016 at 9:42 PM, Charles Srstka via swift-evolution < swift-evolution@swift.org> wrote:

Typo here: This should have said "on the current thread." The fact that
NSProgress bogs down the worker thread with the work involved in creating
the KVO notifications is part of my issue with it.

Charles

On 2016-01-17 21:14, Charles Srstka via swift-evolution wrote:

5. NSProgress posts KVO notifications on the main thread.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Charles Srstka) #8

1. Because NSProgress relies on a thread-local global variable to
store the current progress object, it is impossible to know, outside
of documentation, whether any particular method supports NSProgress
or not, and if the documentation is inadequate, one must resort to
trial-and-error to determine whether NSProgress is supported (this
has been noted before: http://oleb.net/blog/2014/03/nsprogress/
[1]).

iOS 9 and OS X 10.11 support explicit progress composition through the
addChild:withPendingUnitCount: message, and vending of an NSProgress
instance (for use with the addChild:withPendingUnitCount: message)
through the NSProgressReporting protocol. You no longer have to rely
on implicit composition, and in WWDC 2015 session 232 you're
encouraged to use explicit composition.

While this is technically true, 1) it is awkward, bloating method signatures by requiring you to add a "progress:" parameter to all of them, much like the NSError** pointers of old, 2) it doesn't solve the performance issues that NSProgress has with all of its KVO notifications being sent on the worker thread, and 3) you can't even use it, anyway, if you can't require OS X 10.11 or iOS 9.

I have a hard time believing this is something that needs dedicated
language syntax.

It's an incredibly common problem; almost every app that spawns a worker thread is going to need some way to report progress on the task, and a way to cancel it. The more primitive methods are written to support progress reporting, the easier it is for larger operations to report their own progress effectively, so it makes sense to introduce a mechanism that makes progress reporting as easy to implement as possible. I think this is what Apple was hoping for with the original NSProgress implementation; the implicit composition suggests an expectation that all code that does work would support NSProgress, and that you could just expect it. Sadly, that hasn't been the case, but by giving it first-class language support and making it dead easy to use, perhaps we could get closer to that point. At any rate, given Swift's focus on performance (and subsequently, the appropriateness of using pure Swift in a worker thread), it seems a shame to bog everything down with a low-performing progress API like what we have currently.

Charles

···

On 2016-01-18 21:12, Rob Mayoff via swift-evolution wrote:


(Charles Srstka) #9

What do you all think?

I think this is really thin syntactic sugar over `-> NSProgress`, and
doesn't add enough value to justify itself. Unlike the error handling
system:

- The fact that a function *can* provide progress doesn't mean that
any particular caller needs that progress.
- Progress monitoring doesn't require a lot of convoluted control flow.
- Progress monitoring doesn't require convoluted ways of returning the
progress, since most progress-reporting operations are asynchronous
and have no other immediate return value.

This all seems to be under the assumption that NSProgress is an API which is returned to the calling function. This is not how NSProgress works, and in my proposal, progress objects wouldn't be returned; they'd be passed in as a parameter and added as a child to the current progress object. So it's syntactic sugar, yes, but rather than being over -> NSProgress, it's over really quite a lot more than that.

The proposal would reduce this (explicit style):

func doSomethingWithProgress(progress: NSProgress) {
     progress.totalUnitCount = 10

     let childProgress1 = NSProgress()
     progress.addChild(childProgress1, withPendingUnitCount: 5)
     childFunc1WithProgress(childProgress1)

     let childProgress2 = NSProgress()
     progress.addChild(childProgress2, withPendingUnitCount: 5)
     childFunc2WithProgress(childProgress2)
}

func childFunc1WithProgress(progress: NSProgress) {
     progress.totalUnitCount = 10

     for i in 0..<10 {
         // do some work
         progress.completedUnitCount += 1
     }
}

(childFunc2WithProgress looks like childFunc1WithProgress)

or this (implicit style):

func doSomethingWithProgress() {
     let progress = NSProgress(totalUnitCount: 10)

     progress.becomeCurrentWithPendingUnitCount(5)
     childFunc1()
     progress.resignCurrent()

     progress.becomeCurrentWithPendingUnitCount(5)
     childFunc2()
     progress.resignCurrent()
}

func childFunc1() {
     let progress = NSProgress(totalUnitCount: 10)

     for i in 0..<10 {
         progress.completedUnitCount += 1
     }
}

(childFunc2 looks like childFunc1)

to simply this:

func doSomethingWithProgress() reports {
     progress.prepare(10)

     report(5) childFunc1()
     report(5) childFunc2()
}

func childFunc1() reports {
     progress.prepare(something)

     for i in 0..<10 {
         progress.completed += 1
     }
}

(childFunc2 looks like childFunc1)

I don't know about you, but this seems a lot cleaner, and a lot more Swifty, to me. It also combines the safety of the explicit method with the flexibility of the implicit method, while requiring a heck of a lot less code than both.

- Progress monitoring is actually done fairly infrequently.

If the progress object were intelligent and efficient enough to not send notifications on the current thread, the progress object could be updated a lot more than NSProgress is able to. You could optionally apply other optimizations as well, such as coalescing updates rather than firing notifications each time, which would reduce the cost of updating the progress object to a few integer stores and could possibly cut down some more on the code you'd need to clutter your worker loop with.

- There is no need to customize the progress monitoring system in the
way that, for instance, `ErrorType` allows.

I don't know about that; the whole reason I came up with this idea in the first place was because I was wishing I could customize NSProgress in certain ways, the inefficient notification system being chief among them.

I think that all Swift needs to support progress monitoring is to make
sure that `NSProgress` returns can be ignored without a warning even
if we make `@warn_unused_return` the default, and perhaps to overload
`+=` on two `NSProgress`es if we can find an appropriate way to do it.

Again, NSProgress objects are passed in, not returned. += really would have no meaning that I can see for NSProgress.

Charles

···

On 2016-01-19 00:35, Brent Royal-Gordon wrote:


(Charles Srstka) #10

Hi Charles,

Hi Tony,

Thanks very much for responding. I do really appreciate the opportunity to communicate with members of the development team on these lists.

I see Rob already pointed this out, but I’ll respond as well since I
own NSProgress and I also was involved in the WWDC session he
references.

NSProgress actually received a significant upgrade in iOS 9 and OS X
10.11 to address most of your concerns. I’ve never really liked the
‘current’ progress idea either, so I’ve changed Foundation to
make using explicit progress objects possible (and encouraged). There
are still cases where the idea of having a current progress is useful.
For example, when there is no opportunity to grab a progress object,
because the method you are calling is a proxy, or a global function,
etc.).

While this is true, I feel that the syntax for using it is quite verbose, which feels out of place in Swift code. Mainly what it comes down to, though, is that I actually rather like the concept behind what they were trying to do with the "current" progress object, which was to make it possible for all sorts of low-level methods and functions (like -[NSData dataWithContentsOfURL:options:error:]) to automatically and painlessly support progress operations, so that an application that was doing time-consuming operations could get a lot of this "for free." It's just the *implementation* that I quibble with, due to various issues it brings up. I feel like my pitch *works* like the explicit composition method while *feeling* like the implicit composition method, if that makes sense. At any rate, anything that would make progress more attractive to implement, thus increasing its prevalence in the frameworks, would be a plus; I would love to see something like NSFileManager's file copy APIs supporting some sort of progress reporting (in an extension, of course, unless we had some way for Objective-C to call into the Swift progress-reporting mechanism). I also feel like writing it in pure Swift and minimizing the amount of work that is done on the current thread could reduce the performance costs of using it, which is important for these sorts of APIs since you don't want to add unnecessary overhead in the case where you're not interested in monitoring the progress.

You can now create progress trees explicitly. There are two parts to
this:

@available(OSX 10.11, *)
public /*not inherited*/ init(totalUnitCount unitCount: Int64,
parent: NSProgress, pendingUnitCount portionOfParentTotalUnitCount:
Int64)

and a new protocol:

public protocol NSProgressReporting : NSObjectProtocol {
@available(OSX 10.9, *)
public var progress: NSProgress { get }
}

So you can mark classes that publish progress with the protocol, and
clients can create their trees using the new initializer.

This seems oriented toward classes such as NSOperation subclasses that represent a singular operation of some sort, but doesn't seem as applicable to single-function APIs like NSFileManager.copyItemAtURL(toURL:). It can of course be used that way, but it feels somewhat awkward. It also encourages the use of one big monolithic NSProgress object for the whole operation/thread/dispatch queue, which undermines some of what makes NSProgress so cool; the tree-based approach is really great for complex operations which are composed of a lot of separate steps, each of which may have distinct sub-steps of their own.

Yes, agreed. I would encourage the discrete progress if possible, but
I’ve had lots of requests for a block-based version of
becomeCurrent. This is another good reason to add one.

I've been using this rather simple one, which alleviates things quite a bit:

(caveat: written from memory in Mail, as I'm not at my main development machine at the moment)

extension NSProgress {
     public func doWithPendingUnitCount<T>(unitCount: Int64, block: () throws -> T) rethrows -> T {
         self.becomeCurrentWithPendingUnitCount(unitCount)
         defer { self.resignCurrent() }
         return try block()
     }
}

It does not, however, address the performance concerns, nor is it quite as clean (IMO of course) as my suggestion.

I changed the algorithm for how we add child progress objects, so now
we only will add the first one. This wasn’t strictly backwards
compatible, but it was far too easy to add more progress objects than
you intended.

When did this change occur? I feel like I've managed to get the fractionCompleted to go over 1.0 this way on El Capitan, but don't remember exactly how I did it. Good to know this is being worked on, though; if I manage to get this to happen again, I'll file a Radar.

5. NSProgress posts KVO notifications on the main thread.

This actually isn’t true. NSProgress posts KVO notifications on the
thread in which the change was made.

As I mentioned in my followup post written a few minutes after the original one, this was a typo (or, perhaps, a Freudian slip, since posting on the main thread, or more generally on a dispatch queue settable by the user, is what I wish it did). Sorry about that.

In addition to complicating the interface and causing undefined
behavior if one mistakenly binds it to a UI element without
inserting something in between to forward the KVO notifications to
the main thread, this also hurts performance in the worker thread,
since KVO notifications carry a non-trivial performance penalty,
involving not only message sends but also packaging things into an
NSDictionary for the "change" parameter. Notifications for multiple
properties are fired every time completedUnitCount is changed, which
then can cause similar KVO notifications to occur up the family
tree, depending on how many KVO observers are installed. This can
show up as a non-trivial performance cost in Instruments if the
NSProgress object is updated frequently enough.

There is a cost associated with updating progress, it’s true. We can
always do more to optimize it (and we continue to do so), but there
comes a point at which the updater of the progress object has to add
some intelligence of its own. This is why the documentation and
release notes encourage callers to avoid doing things like updating
the progress for each byte in a file (because that is far too many
updates).

It seems to me that a lot of the work for this could be handled by the progress object. Of course, you still wouldn't want to update after every byte, but you wouldn't need to be nearly as vigilant as you do currently. I also feel like it would be better for this to be in pure Swift and to use mechanisms more performant than KVO, since by its very nature this is an API that is intended to run in worker code, which is where you really do care about performance. I realize of course the need for NSProgress to exist, since a Swift-native solution would not be available to pure Objective-C code, but moving forward it would be really nice to have a Swift-native way to do this.

My ideal implementation for the progress objects would probably look something like this:

- The progress object contains some simple integer and boolean properties for the usual things: total units, completed units, and cancelled.

- When one of these properties is changed, the change is propagated to the parent progress for completed units, or to the children for cancelled. This is the only real work that's done on the current thread.

- If and only if someone has set something for at least one of our handlers:
     * Optionally: Check if we've fired the handlers within some threshold, either time based (only fire if we haven't fired in some interval of time) or fraction-based (only fire if the fraction changed has increased enough for us to care about the change), and exit if the threshold hasn't been reached.
     * Fire any handlers that are non-nil, on the dispatch queue which we have as a separate property (defaulting to the main thread). Calculated properties such as fractionCompleted, localizedDescription (in its default implementation), etc. are computed on the dispatch queue as well, rather on the current thread.

This way, the overhead on the current thread when nothing's watching would be limited to a few integer stores, which are fast in Swift, and some nil checks. In the case where we have handlers, we add whatever it costs to run dispatch_async (and some floating-point operations if we implement the coalescing behavior). This should be much more performant than all the dynamic lookups, message sends, change dictionary building, multiple string operations (I'm finding that the KVO notification for localizedDescription seems to get called twice per change when I test it, busting out NSNumberFormatter to rebuild the "X% completed" string each time), and other overhead that we have now.

I could of course end up writing my own implementation, and I may end up doing that. It would be nice to have something like this available as part of the standard distribution, though.

Charles

···

On 2016-01-19 14:01, Tony Parker wrote:


(Dave Abrahams) #11

Introduction:

This is a proposal for a native progress-tracking mechanism in Swift.

<schnipp>

What do you all think?

I think this is really out-of-scope for swift-evolution. There's little
chance we'd want something like this in the standard library in the near
term. It's really Foundation's domain.

My apologies, I didn't read your post all the way down to where you
proposed a language feature. I suggest that you make that clear near
the top of the message and/or in the subject line.

My (slightly more considered) feedback on this proposal is that we're
unlikely to add a language feature *specifically* for progress
reporting. Rather, we'd like to have language features that allow users
to build features like this one as libraries. Some kind of (hygienic)
macro system comes to mind.

-Dave

···

on Wed Jan 20 2016, Dave Abrahams <swift-evolution@swift.org> wrote:

on Sun Jan 17 2016, Charles Srstka via swift-evolution > <swift-evolution-m3FHrko0VLzYtjvyW6yDsg-AT-public.gmane.org> wrote:


(Charles Srstka) #12

While that's true, it hearkens back to those obnoxious NSError ** parameters that had to be put on the end of every last method in ObjC/Cocoa. The Swift do/try/catch mechanism is much nicer, and I was hoping to do something more along those lines. Besides, passing around a single progress would lose NSProgress's greatest feature, which is the tree of progress objects, each of which tracks the progress of a single function or method.

Charles

···

On 2016-01-18 12:26, Will Entriken wrote:

Charles,

I think this is interesting. And discussion on replacing/updating
NSProgress is important.

Just a note: optional parameters make this possible without changing
the language. A very rough example follows:

func delete(files: FileList, withProgress progress: Progress? = nil) {
    progress.totalUnits = 10
    print("Getting started")
    progress.completedUnits = 5 // that was easy!
    actuallyDelete(files, withProgress: progress.childWithUnits: 5)
}

let progress = Progress()
// watch progress.totalPortion using KVO 2.0
delete(files: allTheFiles, withProgress: progress)

Regards,
Will

On Sun, Jan 17, 2016 at 9:42 PM, Charles Srstka via swift-evolution > <swift-evolution@swift.org> wrote:

Typo here: This should have said "on the current thread." The fact
that NSProgress bogs down the worker thread with the work involved
in creating the KVO notifications is part of my issue with it.

Charles

On 2016-01-17 21:14, Charles Srstka via swift-evolution wrote:

5. NSProgress posts KVO notifications on the main thread.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution [1]

Links:
------
[1] https://lists.swift.org/mailman/listinfo/swift-evolution


(Félix Cloutier) #13

In terms of which functions need it, I think that reporting progress is a relatively niche feature. I would by far prefer that we work on infrastructure to implement a library-driven solution. To me, it would be a missed opportunity to show off what the language can (will be able to) do without having to bake it into the compiler.

For instance, this could probably be solved without specific compiler support if Swift had a macro system and resumable functions.

Also, the first two arguments have merit, but it's highly improbable that you'll be able to use whatever solution we come up with in OSes older than OS X 10.11 or iOS 9 anyway.

Félix

···

Le 18 janv. 2016 à 21:59:09, Charles Srstka via swift-evolution <swift-evolution@swift.org> a écrit :

On 2016-01-18 21:12, Rob Mayoff via swift-evolution wrote:

1. Because NSProgress relies on a thread-local global variable to
store the current progress object, it is impossible to know, outside
of documentation, whether any particular method supports NSProgress
or not, and if the documentation is inadequate, one must resort to
trial-and-error to determine whether NSProgress is supported (this
has been noted before: http://oleb.net/blog/2014/03/nsprogress/
[1]).

iOS 9 and OS X 10.11 support explicit progress composition through the
addChild:withPendingUnitCount: message, and vending of an NSProgress
instance (for use with the addChild:withPendingUnitCount: message)
through the NSProgressReporting protocol. You no longer have to rely
on implicit composition, and in WWDC 2015 session 232 you're
encouraged to use explicit composition.

While this is technically true, 1) it is awkward, bloating method signatures by requiring you to add a "progress:" parameter to all of them, much like the NSError** pointers of old, 2) it doesn't solve the performance issues that NSProgress has with all of its KVO notifications being sent on the worker thread, and 3) you can't even use it, anyway, if you can't require OS X 10.11 or iOS 9.

I have a hard time believing this is something that needs dedicated
language syntax.

It's an incredibly common problem; almost every app that spawns a worker thread is going to need some way to report progress on the task, and a way to cancel it. The more primitive methods are written to support progress reporting, the easier it is for larger operations to report their own progress effectively, so it makes sense to introduce a mechanism that makes progress reporting as easy to implement as possible. I think this is what Apple was hoping for with the original NSProgress implementation; the implicit composition suggests an expectation that all code that does work would support NSProgress, and that you could just expect it. Sadly, that hasn't been the case, but by giving it first-class language support and making it dead easy to use, perhaps we could get closer to that point. At any rate, given Swift's focus on performance (and subsequently, the appropriateness of using pure Swift in a worker thread), it seems a shame to bog everything down with a low-performing progress API like what we have currently.

Charles

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution


(Tony Parker) #14

Hi Charles,

Hi Charles,

Hi Tony,

Thanks very much for responding. I do really appreciate the opportunity to communicate with members of the development team on these lists.

Sure, and thanks for your thoughts on this.

I see Rob already pointed this out, but I’ll respond as well since I
own NSProgress and I also was involved in the WWDC session he
references.
NSProgress actually received a significant upgrade in iOS 9 and OS X
10.11 to address most of your concerns. I’ve never really liked the
‘current’ progress idea either, so I’ve changed Foundation to
make using explicit progress objects possible (and encouraged). There
are still cases where the idea of having a current progress is useful.
For example, when there is no opportunity to grab a progress object,
because the method you are calling is a proxy, or a global function,
etc.).

While this is true, I feel that the syntax for using it is quite verbose, which feels out of place in Swift code. Mainly what it comes down to, though, is that I actually rather like the concept behind what they were trying to do with the "current" progress object, which was to make it possible for all sorts of low-level methods and functions (like -[NSData dataWithContentsOfURL:options:error:]) to automatically and painlessly support progress operations, so that an application that was doing time-consuming operations could get a lot of this "for free." It's just the *implementation* that I quibble with, due to various issues it brings up. I feel like my pitch *works* like the explicit composition method while *feeling* like the implicit composition method, if that makes sense. At any rate, anything that would make progress more attractive to implement, thus increasing its prevalence in the frameworks, would be a plus; I would love to see something like NSFileManager's file copy APIs supporting some sort of progress reporting (in an extension, of course, unless we had some way for Objective-C to call into the Swift progress-reporting mechanism). I also feel like writing it in pure Swift and minimizing the amount of work that is done on the current thread could reduce the performance costs of using it, which is important for these sorts of APIs since you don't want to add unnecessary overhead in the case where you're not interested in monitoring the progress.

You can now create progress trees explicitly. There are two parts to
this:
@available(OSX 10.11, *)
public /*not inherited*/ init(totalUnitCount unitCount: Int64,
parent: NSProgress, pendingUnitCount portionOfParentTotalUnitCount:
Int64)
and a new protocol:
public protocol NSProgressReporting : NSObjectProtocol {
@available(OSX 10.9, *)
public var progress: NSProgress { get }
}
So you can mark classes that publish progress with the protocol, and
clients can create their trees using the new initializer.

This seems oriented toward classes such as NSOperation subclasses that represent a singular operation of some sort, but doesn't seem as applicable to single-function APIs like NSFileManager.copyItemAtURL(toURL:). It can of course be used that way, but it feels somewhat awkward. It also encourages the use of one big monolithic NSProgress object for the whole operation/thread/dispatch queue, which undermines some of what makes NSProgress so cool; the tree-based approach is really great for complex operations which are composed of a lot of separate steps, each of which may have distinct sub-steps of their own.

Yes, those cases are why we kept the ‘current’ idea around instead of deprecating it completely. Note that the discrete progress objects still totally embrace the composition pattern that NSProgress is all about. We went into some detail about this in the WWDC presentation.

Yes, agreed. I would encourage the discrete progress if possible, but
I’ve had lots of requests for a block-based version of
becomeCurrent. This is another good reason to add one.

I've been using this rather simple one, which alleviates things quite a bit:

(caveat: written from memory in Mail, as I'm not at my main development machine at the moment)

extension NSProgress {
   public func doWithPendingUnitCount<T>(unitCount: Int64, block: () throws -> T) rethrows -> T {
       self.becomeCurrentWithPendingUnitCount(unitCount)
       defer { self.resignCurrent() }
       return try block()
   }
}

It does not, however, address the performance concerns, nor is it quite as clean (IMO of course) as my suggestion.

I see you filed a bug for this - Thanks for that, and I’ll take it into consideration.

I changed the algorithm for how we add child progress objects, so now
we only will add the first one. This wasn’t strictly backwards
compatible, but it was far too easy to add more progress objects than
you intended.

When did this change occur? I feel like I've managed to get the fractionCompleted to go over 1.0 this way on El Capitan, but don't remember exactly how I did it. Good to know this is being worked on, though; if I manage to get this to happen again, I'll file a Radar.

This change was made for 10.11 and 9.0.

5. NSProgress posts KVO notifications on the main thread.

This actually isn’t true. NSProgress posts KVO notifications on the
thread in which the change was made.

As I mentioned in my followup post written a few minutes after the original one, this was a typo (or, perhaps, a Freudian slip, since posting on the main thread, or more generally on a dispatch queue settable by the user, is what I wish it did). Sorry about that.

In addition to complicating the interface and causing undefined
behavior if one mistakenly binds it to a UI element without
inserting something in between to forward the KVO notifications to
the main thread, this also hurts performance in the worker thread,
since KVO notifications carry a non-trivial performance penalty,
involving not only message sends but also packaging things into an
NSDictionary for the "change" parameter. Notifications for multiple
properties are fired every time completedUnitCount is changed, which
then can cause similar KVO notifications to occur up the family
tree, depending on how many KVO observers are installed. This can
show up as a non-trivial performance cost in Instruments if the
NSProgress object is updated frequently enough.

There is a cost associated with updating progress, it’s true. We can
always do more to optimize it (and we continue to do so), but there
comes a point at which the updater of the progress object has to add
some intelligence of its own. This is why the documentation and
release notes encourage callers to avoid doing things like updating
the progress for each byte in a file (because that is far too many
updates).

It seems to me that a lot of the work for this could be handled by the progress object. Of course, you still wouldn't want to update after every byte, but you wouldn't need to be nearly as vigilant as you do currently. I also feel like it would be better for this to be in pure Swift and to use mechanisms more performant than KVO, since by its very nature this is an API that is intended to run in worker code, which is where you really do care about performance. I realize of course the need for NSProgress to exist, since a Swift-native solution would not be available to pure Objective-C code, but moving forward it would be really nice to have a Swift-native way to do this.

My ideal implementation for the progress objects would probably look something like this:

- The progress object contains some simple integer and boolean properties for the usual things: total units, completed units, and cancelled.

- When one of these properties is changed, the change is propagated to the parent progress for completed units, or to the children for cancelled. This is the only real work that's done on the current thread.

- If and only if someone has set something for at least one of our handlers:
   * Optionally: Check if we've fired the handlers within some threshold, either time based (only fire if we haven't fired in some interval of time) or fraction-based (only fire if the fraction changed has increased enough for us to care about the change), and exit if the threshold hasn't been reached.
   * Fire any handlers that are non-nil, on the dispatch queue which we have as a separate property (defaulting to the main thread). Calculated properties such as fractionCompleted, localizedDescription (in its default implementation), etc. are computed on the dispatch queue as well, rather on the current thread.

This way, the overhead on the current thread when nothing's watching would be limited to a few integer stores, which are fast in Swift, and some nil checks. In the case where we have handlers, we add whatever it costs to run dispatch_async (and some floating-point operations if we implement the coalescing behavior). This should be much more performant than all the dynamic lookups, message sends, change dictionary building, multiple string operations (I'm finding that the KVO notification for localizedDescription seems to get called twice per change when I test it, busting out NSNumberFormatter to rebuild the "X% completed" string each time), and other overhead that we have now.

I could of course end up writing my own implementation, and I may end up doing that. It would be nice to have something like this available as part of the standard distribution, though.

Charles

NSProgress actually has some throttling in place for cross-process progress, and I’ve found it to be very tricky to get right. As far as dropping notifications on the floor, you have to be careful that they aren’t the ones that certain clients care a lot about (e.g., getting to 100% finished). If attempting to limit it to a certain percentage, remember that any given progress object’s idea of a significant percentage may be an insignificant one for the parent progress object - and therefore too frequent. Some clients may really want every progress update and some will want it limited to something like 60Hz.

I don’t think anything about this problem is going to be solved by simply reimplementing the entire thing in Swift. Ditching KVO may seem like a way to increase performance, but then you have to think about the global picture - how is anyone supposed to know when the progress has been updated? You will likely have to invent a different notification mechanism to replace it. At that point you’ve reinvented a chunk of KVO itself instead of focusing on the primary domain of progress reporting. Plus, the notification mechanism we’ve invented is specific to NSProgress and not general enough to plug into other systems effectively.

- Tony

···

On Jan 19, 2016, at 2:24 PM, cocoadev@charlessoft.com wrote:
On 2016-01-19 14:01, Tony Parker wrote:


(Charles Srstka) #15

I suppose I can live with that. Really, my proposal breaks down into two parts; one for a new mechanism, and the other for a nicer syntax for using it. If the language were extended such that the latter could be possible from a library, I guess it’s all the same in the end.

As an aside, do you know if new pure-Swift classes are likely to be added to Foundation, or is that going to remain Objective-C only for the foreseeable future?

Charles

···

On Jan 21, 2016, at 4:09 PM, Dave Abrahams via swift-evolution <swift-evolution@swift.org> wrote:

on Wed Jan 20 2016, Dave Abrahams <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

on Sun Jan 17 2016, Charles Srstka via swift-evolution >> <swift-evolution-m3FHrko0VLzYtjvyW6yDsg-AT-public.gmane.org <http://swift-evolution-m3fhrko0vlzytjvyw6ydsg-at-public.gmane.org/>> wrote:

Introduction:

This is a proposal for a native progress-tracking mechanism in Swift.

<schnipp>

What do you all think?

I think this is really out-of-scope for swift-evolution. There's little
chance we'd want something like this in the standard library in the near
term. It's really Foundation's domain.

My apologies, I didn't read your post all the way down to where you
proposed a language feature. I suggest that you make that clear near
the top of the message and/or in the subject line.

My (slightly more considered) feedback on this proposal is that we're
unlikely to add a language feature *specifically* for progress
reporting. Rather, we'd like to have language features that allow users
to build features like this one as libraries. Some kind of (hygienic)
macro system comes to mind.


(Charles Srstka) #16

In terms of which functions need it, I think that reporting progress
is a relatively niche feature.

How is it niche? Very few apps don't have any places where progress tracking (and user cancellation) would be useful, and it's currently something that has to be done with workarounds that are either clunky or unsafe.

I would by far prefer that we work on
infrastructure to implement a library-driven solution. To me, it would
be a missed opportunity to show off what the language can (will be
able to) do without having to bake it into the compiler.

For instance, this could probably be solved without specific compiler
support if Swift had a macro system and resumable functions.

Apple has repeatedly stated in the past that the lack of macros/a preprocessor is supposed to be a feature, not a bug (cf. https://developer.apple.com/swift/blog/?id=4). I know there have occasionally been some statements on the lists contradicting the official stance, but nonetheless I'm not sure we can count on Swift gaining a macro system. Also, implementing things like this via macros seems a bit... icky to my personal tastes.

Are there any plans to add resumable functions to Swift? I know it's been pitched a few times, but it didn't seem to me like it gained much traction.

Also, the first two arguments have merit, but it's highly improbable
that you'll be able to use whatever solution we come up with in OSes
older than OS X 10.11 or iOS 9 anyway.

All additions to Swift so far have been back-portable to 10.9, given that the runtime gets bundled into the app.

Charles

···

On 2016-01-18 22:24, Félix Cloutier wrote:


(Charles Srstka) #17

NSProgress actually has some throttling in place for cross-process progress, and I’ve found it to be very tricky to get right. As far as dropping notifications on the floor, you have to be careful that they aren’t the ones that certain clients care a lot about (e.g., getting to 100% finished). If attempting to limit it to a certain percentage, remember that any given progress object’s idea of a significant percentage may be an insignificant one for the parent progress object - and therefore too frequent. Some clients may really want every progress update and some will want it limited to something like 60Hz.

My thinking about it is that what the child process considers important doesn’t really matter, because it’s the progress object at the root of the tree that’s going to have listeners on it. When the child progress objects are updated, they will bubble their changes up the tree to the root object, and only there, where the notification handlers are registered. Any coalescing would be done in the code that calls the handlers, and thus, in the common case, in the root object.

The coalescing idea was just an idle thought, however; that’s part of the idea that I haven’t fleshed out yet. It shouldn’t be considered as essential to the idea (which is why I prefaced it with “Optional” in the description of it). If a good implementation of it could be found, though, it could be pretty useful, to clean up and streamline the worker code by removing a bunch of boilerplate that’s not directly related to the work it’s doing.

I don’t think anything about this problem is going to be solved by simply reimplementing the entire thing in Swift. Ditching KVO may seem like a way to increase performance, but then you have to think about the global picture - how is anyone supposed to know when the progress has been updated? You will likely have to invent a different notification mechanism to replace it. At that point you’ve reinvented a chunk of KVO itself instead of focusing on the primary domain of progress reporting. Plus, the notification mechanism we’ve invented is specific to NSProgress and not general enough to plug into other systems effectively.

The Swift way to handle notifications would be simply to have the user register a closure with the progress object. I was thinking putting a few closure properties on the object, called things like progressHandler, cancellationHandler, and so on. These closures could do whatever the developer asks them to; they could update UI elements, send KVO notifications, update a property on some other object that’s bound to a UI element, etc.

KVO has its uses, to be sure; it's great for wiring things up to Interface Builder objects, for instance. However, that advantage is gone in the case of NSProgress, since the notifications don’t fire on the main thread. It’s not safe to update a view object on a secondary thread, making it unsafe to bind something like an NSProgressIndicator directly to a property on NSProgress; you have to use KVO’s awkward observation API in code to catch the notifications and forward them on to the main thread. Meanwhile, the non-trivial costs of KVO are accruing on the worker thread, slowing things down (including things like string ops to calculate properties such as localizedDescription). If things were set up such that the KVO notifications always occurred on the main thread, the value of using KVO here would be obvious, but as it’s implemented, there’s not much benefit to the KVO approach; it's just the wrong tool for the job. All IMO of course.

Charles

···

On Jan 20, 2016, at 2:16 PM, Tony Parker <anthony.parker@apple.com> wrote:


(Dave Abrahams) #18

Sorry, I don't know. You should really be asking Tony that; he's the
Foundation guy.

-Dave

···

on Thu Jan 21 2016, Charles Srstka <cocoadev-AT-charlessoft.com> wrote:

On Jan 21, 2016, at 4:09 PM, Dave Abrahams via swift-evolution <swift-evolution@swift.org> wrote:

on Wed Jan 20 2016, Dave Abrahams <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

on Sun Jan 17 2016, Charles Srstka via swift-evolution > >>> <swift-evolution-m3FHrko0VLzYtjvyW6yDsg-AT-public.gmane.org >>> <http://swift-evolution-m3fhrko0vlzytjvyw6ydsg-at-public.gmane.org/>> >>> wrote:

Introduction:

This is a proposal for a native progress-tracking mechanism in Swift.

<schnipp>

What do you all think?

I think this is really out-of-scope for swift-evolution. There's little
chance we'd want something like this in the standard library in the near
term. It's really Foundation's domain.

My apologies, I didn't read your post all the way down to where you
proposed a language feature. I suggest that you make that clear near
the top of the message and/or in the subject line.

My (slightly more considered) feedback on this proposal is that we're
unlikely to add a language feature *specifically* for progress
reporting. Rather, we'd like to have language features that allow users
to build features like this one as libraries. Some kind of (hygienic)
macro system comes to mind.

I suppose I can live with that. Really, my proposal breaks down into
two parts; one for a new mechanism, and the other for a nicer syntax
for using it. If the language were extended such that the latter could
be possible from a library, I guess it’s all the same in the end.

As an aside, do you know if new pure-Swift classes are likely to be
added to Foundation, or is that going to remain Objective-C only for
the foreseeable future?


(Joe Groff) #19

Lacking the C preprocessor is a feature, lacking a macro system isn't.

-Joe

···

On Jan 18, 2016, at 7:39 PM, Charles Srstka via swift-evolution <swift-evolution@swift.org> wrote:

On 2016-01-18 22:24, Félix Cloutier wrote:

In terms of which functions need it, I think that reporting progress
is a relatively niche feature.

How is it niche? Very few apps don't have any places where progress tracking (and user cancellation) would be useful, and it's currently something that has to be done with workarounds that are either clunky or unsafe.

I would by far prefer that we work on
infrastructure to implement a library-driven solution. To me, it would
be a missed opportunity to show off what the language can (will be
able to) do without having to bake it into the compiler.
For instance, this could probably be solved without specific compiler
support if Swift had a macro system and resumable functions.

Apple has repeatedly stated in the past that the lack of macros/a preprocessor is supposed to be a feature, not a bug (cf. https://developer.apple.com/swift/blog/?id=4). I know there have occasionally been some statements on the lists contradicting the official stance, but nonetheless I'm not sure we can count on Swift gaining a macro system. Also, implementing things like this via macros seems a bit... icky to my personal tastes.


(Félix Cloutier) #20

In terms of which functions need it, I think that reporting progress
is a relatively niche feature.

How is it niche? Very few apps don't have any places where progress tracking (and user cancellation) would be useful, and it's currently something that has to be done with workarounds that are either clunky or unsafe.

It's niche in that even if most applications will have functions that use it, most functions don't need it.

I would by far prefer that we work on
infrastructure to implement a library-driven solution. To me, it would
be a missed opportunity to show off what the language can (will be
able to) do without having to bake it into the compiler.
For instance, this could probably be solved without specific compiler
support if Swift had a macro system and resumable functions.

Apple has repeatedly stated in the past that the lack of macros/a preprocessor is supposed to be a feature, not a bug (cf. https://developer.apple.com/swift/blog/?id=4). I know there have occasionally been some statements on the lists contradicting the official stance, but nonetheless I'm not sure we can count on Swift gaining a macro system. Also, implementing things like this via macros seems a bit... icky to my personal tastes.

Are there any plans to add resumable functions to Swift? I know it's been pitched a few times, but it didn't seem to me like it gained much traction.

As far as I can tell, support for the feature is positive; except that the core team said "not yet <https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151221/004158.html>". I, for sure, will be on the yes side when we get there.

For the macro system, I'm talking about a system that lets you modify Swift code at compile-time. An example system would be to allow Swift code to modify an AST. I'm absolutely not an expert in this domain, I hope a LISPy person will write a proposal about it!

Also, the first two arguments have merit, but it's highly improbable
that you'll be able to use whatever solution we come up with in OSes
older than OS X 10.11 or iOS 9 anyway.

All additions to Swift so far have been back-portable to 10.9, given that the runtime gets bundled into the app.

I'm not at Apple, but my gut feeling is that it won't be possible for much longer, given that Swift 3 should stabilize the ABI and make bundling unnecessary.

Félix

···

Le 18 janv. 2016 à 22:39:19, cocoadev@charlessoft.com a écrit :
On 2016-01-18 22:24, Félix Cloutier wrote: