Pitch: @required attribute for closures

Responses inline

Sent from my iPad

Thanks Matthew, my responses are inline:

Sent from my iPad

Perhaps I was unclear, in my explanation. The guarantee I'm enforcing is
that the closure is called exactly once before being released.

Everything I suggested is a compile-time check.

The compile-time warning and runtime `fatalError` I suggested could be
replaced with a compile-time error, however even in this case it is still
statically checked for the warning.

The compiler can statically guarantee *exactly one* of these things
happens in methods using the closure:

   - the closure is *called*
   - the closure is *stored*
   - the closure is *passed* to another method
   - the program *aborts* with something like a fatalError

If the closure is stored then there must be a *deinit*, and those checks
apply there as well.

I believe this is sufficient to ensure the closure is called once. Please
let me know if there are any cases these checks miss.

If the closure is stored in a member it could be called in the
implementation of any other member. Calls to other members could come from
arbitrary locations in the surrounding program at arbitrary points in time
(unless you have static analysis that can prove a narrower set of
possibilities).

This isn't a problem if the member has to have the type annotation, all
uses of the member, whether from elsewhere in the program, or other
methods, would have to pass the checks.

If you call the closure you must nil/replace the member.

Ok, so the member must be an optional? If that is the idea I would
suggest considering something similar to 'weak' where it automatically gets
set to nil after it is called (which could maybe become a property behavior
in the future).

That's a great idea, auto-nilling would work well. I was also thinking it
should be possible to replace an existing non-optional stored closure if
you call the old one in the old scope (see the callClosure method in my
previous post). However auto-nilling could simplify the concept.

I don't think you mentioned the case of reassigning the member when it is
non-nil. You would have to require users to verify it is nil before
setting it or if it is not nil, calling it before assigning to it.

And if you have a model that relies on behavior in a deinit then storing

the closure won't be possible for structs.

This is true, considering you don't want to copy a @once closure you
probably don't want value-type semantics anyway.

That's a good point. Since we don't have control over copy behavior in
Swift it wouldn't make sense at all unless / until we can make structs that
have move semantics (maybe if / when we get a Rust-like ownership system?).

Interesting, thanks, I wasn't aware of Rust's ownership concept - I've been
working on a related proposal for Swift.

You have also missed the case that the closure is captured by another

closure (maybe it is a completion block and you call it in a completion
block of a method your method calls).

This is correct. I forgot to mention that I'm sorry, thanks for pointing
it out!

I was thinking that a @once closure can only be captured by another @once
closure. We can add that as another dot-point:

   - the closure is captured by another @once closure, this is the only
   time it can be captured.

I like the idea behind this proposal in theory. However, it really seems
to cry out for linear types. I have a feeling we would end up with a
better (and more general) solution if Swift goes down that path in the
future. At minimum, I would like to hear input from those who work on the
type system. If a more robust, less ad-hoc solution will be possible in
the future it might be best to wait.

On the other hand, completion callbacks that require this guarantee are
pretty common. The semantic is part of the contract whether we have
language support for it or not. Maybe we can do something now that could
be subsumed by a more general feature in the future...

I have the same thinking, I'd like the semantics now, and as as strongly

as the language can allow. If the language gets better type support then we
get better or less restrictive assurances.

···

On Mon, Jun 6, 2016 at 12:23 PM, Matthew Johnson <matthew@anandabits.com> wrote:

On Jun 5, 2016, at 8:43 PM, Andrew Bennett <cacoyi@gmail.com> wrote:
On Mon, Jun 6, 2016 at 10:32 AM, Matthew Johnson <matthew@anandabits.com> > wrote:

On Jun 5, 2016, at 6:50 PM, Andrew Bennett <cacoyi@gmail.com> wrote:

On Sun, Jun 5, 2016 at 11:59 PM, Matthew Johnson <matthew@anandabits.com> >> wrote:

Sent from my iPad

On Jun 5, 2016, at 8:52 AM, Andrew Bennett <cacoyi@gmail.com> wrote:

Storing into a member would be fine, as long as it must keep @once as a
type annotation and the compiler makes sure you maintain:
    sum(callCount, storeCount, passCount) == 1

For example:
  class Example {
    private var closure: (@once (T) -> Void)?

    func callClosure(value: T, replace: (@once (T) -> Void)? = nil) {

      // the compiler should error if it detects the closure:

      // * escaping more than once, while still being stored,

      // * or being called while still being stored or escaping,

      // * or being overwritten without being called

      if let closure = self.closure {

        self.closure = replace

        closure(value)

      }

    }

    deinit {

      // compiler warning: that closure is potentially un-called

      // runtime fatalError if it's .Some(Closure) after deinit

    }

  }

There could be a standard library type with those guarantees built in.

I don't consider this compiler verification. It is runtime
verification. The best the compiler can do is enforce constraints that
allow for guaranteed runtime verification. You can argue that is better
than nothing but it is not a static guarantee of correct behavior.

On Sun, Jun 5, 2016 at 10:12 PM, Matthew Johnson <matthew@anandabits.com >>> > wrote:

Sent from my iPad

On Jun 5, 2016, at 6:56 AM, Andrew Bennett <cacoyi@gmail.com> wrote:

I like this.

One of the suggestions on @noescape(once) was that it just becomes
@once and works with escaping closures too. It might be possible if compile
time checks verified that the closure isn't copied, and that it is called
before being deinit-ialized. Failing that I'm happy with a runtime
circumstance in the cases the compiler can't check.

Yeah, maybe if it is only used asynchronously and never stored in a
member or global it could be verified and that is a pretty common case.
That would certainly be easier than the general case.

I prefer @once over @required if the guarantee is single execution. If
the guarantee is *at least once* obviously @once is not the right
attribute, but I'm not convinced @required is either. Maybe @invoked.

It would be great if @required took into the account the feedback from
that proposal and considered the synchronous case too.

As an aside, you can get some of the guarantees you want like this:

func doSomething(completionHandler: (SomeEnum) -> ()) {

  dispatch_async(someQueue) {

    let result: SomeEnum

    // the compiler ensures 'result' is set

    defer { completionHandler(result) }

    if aCondition {

      if bCondition {

        result = .Foo

      } else {

        result = .Bar

      }

      // the compiler ensures you do this, because it is 'let'

      return
    }

    if cCondition {

      result = .Baz

    }

  }

}

On Sun, Jun 5, 2016 at 9:42 PM, Matthew Johnson via swift-evolution < >>>> swift-evolution@swift.org> wrote:

Sent from my iPad

On Jun 5, 2016, at 5:02 AM, Patrick Pijnappel via swift-evolution < >>>>> swift-evolution@swift.org> wrote:

This has actually been proposed before, see SE-0073:
https://github.com/apple/swift-evolution/blob/master/proposals/0073-noescape-once.md

Actually that proposal was for noescape closures and this suggestion
is for escaping closures. I don't think the compiler can verify this for
noescape closures. If it is possible it would be far more complicated.

On Sun, Jun 5, 2016 at 11:37 AM, Charles Srstka via swift-evolution < >>>>> swift-evolution@swift.org> wrote:

MOTIVATION:

As per the current situation, there is a pitfall when writing
asynchronous APIs that does not occur when writing synchronous APIs.
Consider the following synchronous API:

func doSomething() -> SomeEnum {
        if aCondition {
                if bCondition {
                        return .Foo
                } else {
                        return .Bar
                }
        } else {
                if cCondition {
                        return .Baz
                }
        }
}

The compiler will give an error here, since if both aCondition and
cCondition are false, the function will not return anything.

However, consider the equivalent async API:

func doSomething(completionHandler: (SomeEnum) -> ()) {
        dispatch_async(someQueue) {
                if aCondition {
                        if bCondition {
                                completionHandler(.Foo)
                        } else {
                                completionHandler(.Bar)
                        }
                } else {
                        if cCondition {
                                completionHandler(.Baz)
                        }
                }
        }
}

Whoops, now the function can return without ever firing its
completion handler, and the problem might not be discovered until runtime
(and, depending on the complexity of the function, may be hard to find).

PROPOSED SOLUTION:

Add a @required attribute that can be applied to closure arguments.
This attribute simply states that the given closure will always be
eventually called, and the compiler can enforce this.

DETAILED DESIGN:

- The @required attribute states in our API contract that a given
closure *must* be called at some point after the function is called.

- Standard API calls like dispatch_async that contractually promise
to execute a closure or block get @required added to their signatures.

- When the compiler sees a @required closure in a function
declaration, it checks to make sure that every execution path either calls
the closure at some point, or sends a @required closure to another API that
eventually ends up calling the closure.

- If there’s a way for a @required closure not to be called, the
compiler emits an error letting the developer know about the bug in his/her
code.

IMPACT ON EXISTING CODE:

None. This is purely additive.

ALTERNATIVES CONSIDERED:

I got nothin’.

Charles

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Yes, ok...

Well, I cannot easily make everything synchronous. I just can change it in a way that makes a `@required` attribute "optional" ;)

import Foundation

enum Result<T> {
   case success(T)
   case error(ErrorType)
}

enum MyError: ErrorType {
   case unknownError
   case corruptData
   case badStatusCode(Int)
}

struct SomeThing {
   init?(data: NSData) {
       ...
   }
}

func getSomethingFromTheNetwork(url: NSURL, completionHandler: (Result<SomeThing>) -> ()) {
   func toSomeThing(data: NSData?, response: NSURLResponse?, error: NSError?) -> Result<SomeThing> {
       if let error = error {
           return .error(error)
       }

       if let httpResponse = response as? NSHTTPURLResponse {
           let statusCode = httpResponse.statusCode

           if statusCode < 200 || statusCode >= 300 {
               return .error(MyError.badStatusCode(statusCode))
           }
       }

       guard let data = data else {
           return .error(MyError.unknownError)
       }

       guard let something = SomeThing(data: data) else {
           return .error(MyError.corruptData)
       }

       return .success(something)
   }
   let task = NSURLSession.sharedSession().dataTaskWithURL(url) { data, response, error in
       completionHandler(toSomething(data, response, error))
   }

   task.resume()
}

I’ve written code like that; however, it falls down as soon as you have to nest two or more asynchronous APIs. Consider the following, and assume we don’t have control of SomeThing’s API (or, maybe, SomeThing relies on some other API we don’t control that has to be asynchronous):

import Foundation

enum Result<T> {
   case success(T)
   case error(ErrorType)
}

enum MyError: ErrorType {
   case unknownError
   case corruptData
   case badStatusCode(Int)
}

struct SomeThing {
   make(data: NSData, completionHandler: (SomeThing?) -> ())
}

func getSomethingFromTheNetwork(url: NSURL, completionHandler: (Result<SomeThing>) -> ()) {
  let task = NSURLSession.sharedSession().dataTaskWithURL(url) { data, response, error in
    if let error = error {
      completionHandler(.error(error))
      return
    }

    if let httpResponse = response as? NSHTTPURLResponse {
      let statusCode = httpResponse.statusCode
      
      if statusCode < 200 || statusCode >= 300 {
        completionHandler(.error(MyError.badStatusCode(statusCode)))
        return
      }
    }
  
    guard let data = data else {
      completionHandler(.error(MyError.unknownError))
      return
    }

    SomeThing.make(data) { something in
      if let something = something {
        completionHandler(.success(something))
      } else {
        completionHandler(.error(MyError.corruptData))
      }
    }
  }
  
  task.resume()
}

Yeah, you can wrap it in yet another function, but if things get complex enough, and you forget to do that somewhere, it’s bad news.

With a semaphore, I can also make it synchronous. Not sure if this is a good idea though... If the API is already asynchronous, it's probably better to use it that way.

It’s not a good idea. As I mentioned before, Apple recommends against blocking on dispatch queues. It’s not considered good practice. That includes the main queue, by the way; both Cocoa-Dev and Stack Overflow are filled with questions from people who tried using dispatch_sync or dispatch_async to the main queue to run -[NSOpenPanel runModal], and then got confused when nothing in the panel displayed properly. Using the async methods on NSOpenPanel and returning via a completion handler would have caused this to work properly.

Charles

···

On Jun 6, 2016, at 3:48 PM, michael.peternell@gmx.at wrote:

Responses inline

Sent from my iPad

Thanks Matthew, my responses are inline:

Sent from my iPad

Perhaps I was unclear, in my explanation. The guarantee I'm enforcing is that the closure is called exactly once before being released.

Everything I suggested is a compile-time check.

The compile-time warning and runtime `fatalError` I suggested could be replaced with a compile-time error, however even in this case it is still statically checked for the warning.

The compiler can statically guarantee exactly one of these things happens in methods using the closure:
the closure is called
the closure is stored
the closure is passed to another method
the program aborts with something like a fatalError
If the closure is stored then there must be a deinit, and those checks apply there as well.

I believe this is sufficient to ensure the closure is called once. Please let me know if there are any cases these checks miss.

If the closure is stored in a member it could be called in the implementation of any other member. Calls to other members could come from arbitrary locations in the surrounding program at arbitrary points in time (unless you have static analysis that can prove a narrower set of possibilities).

This isn't a problem if the member has to have the type annotation, all uses of the member, whether from elsewhere in the program, or other methods, would have to pass the checks.

If you call the closure you must nil/replace the member.

Ok, so the member must be an optional? If that is the idea I would suggest considering something similar to 'weak' where it automatically gets set to nil after it is called (which could maybe become a property behavior in the future).

That's a great idea, auto-nilling would work well. I was also thinking it should be possible to replace an existing non-optional stored closure if you call the old one in the old scope (see the callClosure method in my previous post). However auto-nilling could simplify the concept.

I don't think you mentioned the case of reassigning the member when it is non-nil. You would have to require users to verify it is nil before setting it or if it is not nil, calling it before assigning to it.

And if you have a model that relies on behavior in a deinit then storing the closure won't be possible for structs.

This is true, considering you don't want to copy a @once closure you probably don't want value-type semantics anyway.

That's a good point. Since we don't have control over copy behavior in Swift it wouldn't make sense at all unless / until we can make structs that have move semantics (maybe if / when we get a Rust-like ownership system?).
Interesting, thanks, I wasn't aware of Rust's ownership concept - I've been working on a related proposal for Swift.

Rust is really leading the way here in bringing linear types into a mainstream language. Getting it right isn’t easy. Rust’s system has evolved quite a bit over the years since they began. It will be good to learn as much as possible from Rust (both what it gets right and also areas that could maybe be improved).

If you have a draft of anything and would like feedback feel free to send me a link (on or off list). I have a lot of interest in this area as well.

···

On Jun 5, 2016, at 9:40 PM, Andrew Bennett <cacoyi@gmail.com> wrote:
On Mon, Jun 6, 2016 at 12:23 PM, Matthew Johnson <matthew@anandabits.com <mailto:matthew@anandabits.com>> wrote:
On Jun 5, 2016, at 8:43 PM, Andrew Bennett <cacoyi@gmail.com <mailto:cacoyi@gmail.com>> wrote:

On Mon, Jun 6, 2016 at 10:32 AM, Matthew Johnson <matthew@anandabits.com <mailto:matthew@anandabits.com>> wrote:
On Jun 5, 2016, at 6:50 PM, Andrew Bennett <cacoyi@gmail.com <mailto:cacoyi@gmail.com>> wrote:

You have also missed the case that the closure is captured by another closure (maybe it is a completion block and you call it in a completion block of a method your method calls).

This is correct. I forgot to mention that I'm sorry, thanks for pointing it out!

I was thinking that a @once closure can only be captured by another @once closure. We can add that as another dot-point:

the closure is captured by another @once closure, this is the only time it can be captured.

I like the idea behind this proposal in theory. However, it really seems to cry out for linear types. I have a feeling we would end up with a better (and more general) solution if Swift goes down that path in the future. At minimum, I would like to hear input from those who work on the type system. If a more robust, less ad-hoc solution will be possible in the future it might be best to wait.

On the other hand, completion callbacks that require this guarantee are pretty common. The semantic is part of the contract whether we have language support for it or not. Maybe we can do something now that could be subsumed by a more general feature in the future...

I have the same thinking, I'd like the semantics now, and as as strongly as the language can allow. If the language gets better type support then we get better or less restrictive assurances.

On Sun, Jun 5, 2016 at 11:59 PM, Matthew Johnson <matthew@anandabits.com <mailto:matthew@anandabits.com>> wrote:

Sent from my iPad

On Jun 5, 2016, at 8:52 AM, Andrew Bennett <cacoyi@gmail.com <mailto:cacoyi@gmail.com>> wrote:

Storing into a member would be fine, as long as it must keep @once as a type annotation and the compiler makes sure you maintain:
    sum(callCount, storeCount, passCount) == 1

For example:
  class Example {
    private var closure: (@once (T) -> Void)?

    func callClosure(value: T, replace: (@once (T) -> Void)? = nil) {
      // the compiler should error if it detects the closure:
      // * escaping more than once, while still being stored,
      // * or being called while still being stored or escaping,
      // * or being overwritten without being called
      if let closure = self.closure {
        self.closure = replace
        closure(value)
      }
    }

    deinit {
      // compiler warning: that closure is potentially un-called
      // runtime fatalError if it's .Some(Closure) after deinit
    }
  }

There could be a standard library type with those guarantees built in.

I don't consider this compiler verification. It is runtime verification. The best the compiler can do is enforce constraints that allow for guaranteed runtime verification. You can argue that is better than nothing but it is not a static guarantee of correct behavior.

On Sun, Jun 5, 2016 at 10:12 PM, Matthew Johnson <matthew@anandabits.com <mailto:matthew@anandabits.com>> wrote:

Sent from my iPad

On Jun 5, 2016, at 6:56 AM, Andrew Bennett <cacoyi@gmail.com <mailto:cacoyi@gmail.com>> wrote:

I like this.

One of the suggestions on @noescape(once) was that it just becomes @once and works with escaping closures too. It might be possible if compile time checks verified that the closure isn't copied, and that it is called before being deinit-ialized. Failing that I'm happy with a runtime circumstance in the cases the compiler can't check.

Yeah, maybe if it is only used asynchronously and never stored in a member or global it could be verified and that is a pretty common case. That would certainly be easier than the general case.

I prefer @once over @required if the guarantee is single execution. If the guarantee is *at least once* obviously @once is not the right attribute, but I'm not convinced @required is either. Maybe @invoked.

It would be great if @required took into the account the feedback from that proposal and considered the synchronous case too.

As an aside, you can get some of the guarantees you want like this:

func doSomething(completionHandler: (SomeEnum) -> ()) {
  dispatch_async(someQueue) {
    let result: SomeEnum
    // the compiler ensures 'result' is set
    defer { completionHandler(result) }

    if aCondition {
      if bCondition {
        result = .Foo
      } else {
        result = .Bar
      }
      // the compiler ensures you do this, because it is 'let'
      return
    }

    if cCondition {
      result = .Baz
    }
  }
}

On Sun, Jun 5, 2016 at 9:42 PM, Matthew Johnson via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Sent from my iPad

On Jun 5, 2016, at 5:02 AM, Patrick Pijnappel via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

This has actually been proposed before, see SE-0073: https://github.com/apple/swift-evolution/blob/master/proposals/0073-noescape-once.md

Actually that proposal was for noescape closures and this suggestion is for escaping closures. I don't think the compiler can verify this for noescape closures. If it is possible it would be far more complicated.

On Sun, Jun 5, 2016 at 11:37 AM, Charles Srstka via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
MOTIVATION:

As per the current situation, there is a pitfall when writing asynchronous APIs that does not occur when writing synchronous APIs. Consider the following synchronous API:

func doSomething() -> SomeEnum {
        if aCondition {
                if bCondition {
                        return .Foo
                } else {
                        return .Bar
                }
        } else {
                if cCondition {
                        return .Baz
                }
        }
}

The compiler will give an error here, since if both aCondition and cCondition are false, the function will not return anything.

However, consider the equivalent async API:

func doSomething(completionHandler: (SomeEnum) -> ()) {
        dispatch_async(someQueue) {
                if aCondition {
                        if bCondition {
                                completionHandler(.Foo)
                        } else {
                                completionHandler(.Bar)
                        }
                } else {
                        if cCondition {
                                completionHandler(.Baz)
                        }
                }
        }
}

Whoops, now the function can return without ever firing its completion handler, and the problem might not be discovered until runtime (and, depending on the complexity of the function, may be hard to find).

PROPOSED SOLUTION:

Add a @required attribute that can be applied to closure arguments. This attribute simply states that the given closure will always be eventually called, and the compiler can enforce this.

DETAILED DESIGN:

- The @required attribute states in our API contract that a given closure *must* be called at some point after the function is called.

- Standard API calls like dispatch_async that contractually promise to execute a closure or block get @required added to their signatures.

- When the compiler sees a @required closure in a function declaration, it checks to make sure that every execution path either calls the closure at some point, or sends a @required closure to another API that eventually ends up calling the closure.

- If there’s a way for a @required closure not to be called, the compiler emits an error letting the developer know about the bug in his/her code.

IMPACT ON EXISTING CODE:

None. This is purely additive.

ALTERNATIVES CONSIDERED:

I got nothin’.

Charles

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution