Proposal: Add generator functions to the language

I really like this form…

    func countToTen() -> yields Int { yield 1; yield 2 }

It reads well and states clearly what it’s intent is.

Works/reads well with ‘throws’...

    func countToTen() throws -> yields Int {}

And there must be some advantage to claiming two keywords that are nearly identical, ‘yield’ & ‘yields’.

···

On Dec 11, 2015, at 8:45 PM, Alex Gordon via swift-evolution <swift-evolution@swift.org> wrote:

IMO the yields should be literally *in* the return type. The present state of things is

fn countToTen() -> CountToTenGenerator

Generator functions are normal functions. What is special about them is that instead of returning a named type such as "CountToTenGenerator", the compiler replaces the return type with an anonymous type conforming to GeneratorType. So conceptually it's the return type that is special, not the function.

Try either one of these for size

fn countToTen() -> yields Int {
fn countToTen() -> yields<Int> {

- Alex

On Sat, Dec 12, 2015 at 3:17 AM, Joe Groff via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Dec 11, 2015, at 6:26 PM, Jordan Rose via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Eh, I was trying to avoid grabbing another keyword, but I guess it's context-sensitive anyway.

I've thought about this some. It might not have to be a keyword, if this were a generalized language feature. Anything that interrupts control flow and optionally resumes it later, such as 'throws', 'yields', and potentially also 'async', could be implemented as instances of algebraic effects. As a rough sketch of an idea, you could declare an effect and its operations:

effect throws { @noreturn operation throw (ErrorType) -> () }
effect yields<T> { operation yield (T) -> () }
effect awaits { operation await<T> (Promise<T>) -> T }

and 'catch' could be generalized to let you handle any effect operations that might be performed in the body of a block:

class Generator<T> {
  var generator: () yields<T> -> ()
  func next() -> T? {
    do {
        generator()
      return nil
    } catch yield (let x) {
      generator = currentContinuation
      return x
    }
  }
}

See Eff (http://www.eff-lang.org <http://www.eff-lang.org/&gt;\) for an example of a language with this already implemented.

-Joe

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

Another possible syntax, inspired by "throws".

func foo() yield -> String

Yes, I also suspect that would play better with generators from closures.

I'm not sure what happens if a function both yields and throws. I guess that can't happen right now because generators are required not to throw when producing their next() value.

The only way I’ve thought of so far is to have the next iteration return Either a correct value or an ErrorType - probably through a Result enum. Unlike Optional however, there doesn’t seem to be a monadic way to represent possibly errored states - you are meant to use do/try/throw/catch.

-DW

···

On Dec 11, 2015, at 7:18 PM, Jordan Rose via swift-evolution <swift-evolution@swift.org> wrote:

Jordan

On Dec 11, 2015, at 18:14 , Jordan Rose <jordan_rose@apple.com <mailto:jordan_rose@apple.com>> wrote:

It's desireable for the signature of a function to be fully checkable without looking at the body, for a number of reasons. It's also something clients of a library will want to know, so it needs to be displayed in the library's interface somehow.

Jordan

On Dec 11, 2015, at 18:03 , Andrey Tarantsov via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Do we actually need the "generator" keyword? Why can't the compiler figure it out by the presence of yield?

And "yield" is a fairly standard keyword across languages, so no harm in claiming it, IMO.

+1 on the proposal.

A.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

Shouldn't it be:

func foo() yields -> String

Also +1 on the idea.

/Jonathan

···

12 dec. 2015 kl. 03:18 skrev Jordan Rose via swift-evolution <swift-evolution@swift.org>:

Another possible syntax, inspired by "throws".

func foo() yield -> String

I'm not sure what happens if a function both yields and throws. I guess that can't happen right now because generators are required not to throw when producing their next() value.

Jordan

On Dec 11, 2015, at 18:14 , Jordan Rose <jordan_rose@apple.com> wrote:

It's desireable for the signature of a function to be fully checkable without looking at the body, for a number of reasons. It's also something clients of a library will want to know, so it needs to be displayed in the library's interface somehow.

Jordan

On Dec 11, 2015, at 18:03 , Andrey Tarantsov via swift-evolution <swift-evolution@swift.org> wrote:

Do we actually need the "generator" keyword? Why can't the compiler figure it out by the presence of yield?

And "yield" is a fairly standard keyword across languages, so no harm in claiming it, IMO.

+1 on the proposal.

A.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

IMO the yields should be literally *in* the return type. The present state
of things is

fn countToTen() -> CountToTenGenerator

Generator functions are normal functions. What is special about them is
that instead of returning a named type such as "CountToTenGenerator", the
compiler replaces the return type with an anonymous type conforming to
GeneratorType. So conceptually it's the return type that is special, not
the function.

Try either one of these for size

fn countToTen() -> yields Int {
fn countToTen() -> yields<Int> {

- Alex

···

On Sat, Dec 12, 2015 at 3:17 AM, Joe Groff via swift-evolution < swift-evolution@swift.org> wrote:

On Dec 11, 2015, at 6:26 PM, Jordan Rose via swift-evolution < > swift-evolution@swift.org> wrote:

Eh, I was trying to avoid grabbing another keyword, but I guess it's
context-sensitive anyway.

I've thought about this some. It might not have to be a keyword, if this
were a generalized language feature. Anything that interrupts control flow
and optionally resumes it later, such as 'throws', 'yields', and
potentially also 'async', could be implemented as instances of algebraic
effects. As a rough sketch of an idea, you could declare an effect and its
operations:

effect throws { @noreturn operation throw (ErrorType) -> () }
effect yields<T> { operation yield (T) -> () }
effect awaits { operation await<T> (Promise<T>) -> T }

and 'catch' could be generalized to let you handle any effect operations
that might be performed in the body of a block:

class Generator<T> {
var generator: () yields<T> -> ()
func next() -> T? {
do {
   generator()
return nil
} catch yield (let x) {
generator = currentContinuation
return x
}
}
}

See Eff (http://www.eff-lang.org) for an example of a language with this
already implemented.

-Joe

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

I think you need some explicit marker. Consider the following:

   func f() -> Int {
       return
   }

Is this (1) a generator that returns without yielding anything, or (2) an ordinary function that incorrectly fails to return a value?

Perhaps that's too much magic, and generator functions should explicitly declare the return type of GeneratorType<Xxx>. Being a generator is an implementation detail, after all.

Also, I'm not sure than a statically empty generator function is a useful concept. You could just say that without a yield, it's not a generator, period.

A.

Another possible syntax, inspired by "throws".

func foo() yield -> String

Well, again, yielding is an implementation detail. What's the advantage vs returning Generator<String>? I don't think that lying about the return type is useful for anyone.

I'm not sure what happens if a function both yields and throws.

An excellent question. We'd need another generator type with "next() throws".

A.

This feature (set) as sketched here by Joe is worthwhile, in my opinion.
The use of catch fits in and effect neatly captures this class of behaviors.
+1 for `effect`

···

---
TJ

On Sat, Dec 12, 2015 at 4:17 AM, Joe Groff via swift-evolution < swift-evolution@swift.org> wrote:

On Dec 11, 2015, at 6:26 PM, Jordan Rose via swift-evolution < > swift-evolution@swift.org> wrote:

Eh, I was trying to avoid grabbing another keyword, but I guess it's
context-sensitive anyway.

I've thought about this some. It might not have to be a keyword, if this
were a generalized language feature. Anything that interrupts control flow
and optionally resumes it later, such as 'throws', 'yields', and
potentially also 'async', could be implemented as instances of algebraic
effects. As a rough sketch of an idea, you could declare an effect and its
operations:

effect throws { @noreturn operation throw (ErrorType) -> () }
effect yields<T> { operation yield (T) -> () }
effect awaits { operation await<T> (Promise<T>) -> T }

and 'catch' could be generalized to let you handle any effect operations
that might be performed in the body of a block:

class Generator<T> {
var generator: () yields<T> -> ()
func next() -> T? {
do {
   generator()
return nil
} catch yield (let x) {
generator = currentContinuation
return x
}
}
}

See Eff (http://www.eff-lang.org) for an example of a language with this
already implemented.

-Joe

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Is there a possibility for larger features like this or macros to be done

in parallel with the 2.2/3.0 development effort? Or perhaps a desire to
have this sort of working group approach (similar to C++’s Technical
Subcommittees), but not until 3.0 is released?

It's open source! Just make a public fork on GitHub, then other people can
contribute if they desire. Discussion can take place on a separate GitHub
issues if it would be too speculative for this list (can anything be too
speculative for this list?).

The main problem right now seems to be that nobody has any clue how to even
implement it.

- Alex

···

On Sat, Dec 12, 2015 at 9:36 PM, David Waite via swift-evolution < swift-evolution@swift.org> wrote:

I proposed it because
- Only a few named language features (like Concurrency) were explicit
non-goals
- Its a feature I’m willing to try to implement (although I would probably
want to work on several smaller changes to get up to speed first on SIL and
the codebase)

Is there a possibility for larger features like this or macros to be done
in parallel with the 2.2/3.0 development effort? Or perhaps a desire to
have this sort of working group approach (similar to C++’s Technical
Subcommittees), but not until 3.0 is released?

-DW

On Dec 12, 2015, at 1:35 PM, Douglas Gregor via swift-evolution < > swift-evolution@swift.org> wrote:

On Dec 11, 2015, at 10:48 PM, Kevin Ballard via swift-evolution < > swift-evolution@swift.org> wrote:

I'd love to have first-class support for generators like this, but it's a
*lot* of work. It's a lot easier to do this sort of thing in a scripting
language like Python than it is to do in a language like Swift, because it
requires reifying the stack into a data structure that can be passed
around. And I suspect there's a lot of non-trivial questions that have to
get answered before you can even propose an implementation for this.

For context, this sort of thing is something that people have been talking
about doing in Rust for quite a while, and it keeps getting punted because
of the amount of work and the unanswered questions about how it would
actually be implemented.

So I'll give this a general +1, but I think it should also be deferred
until after Swift 3 at the earliest.

Right. This is a major add-on feature that doesn’t fit in with the stated
goals for Swift 3 (README of https://github.com/apple/swift-evolution\),
so I think we should proactively defer it.

- Doug

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

I proposed it because
- Only a few named language features (like Concurrency) were explicit non-goals
- Its a feature I’m willing to try to implement (although I would probably want to work on several smaller changes to get up to speed first on SIL and the codebase)

Is there a possibility for larger features like this or macros to be done in parallel with the 2.2/3.0 development effort?

Yes, design and experimental implementation can certainly proceed on the side. However, we are bandwidth-constrained on how much design we can do, so (for example) the core team may not be able to engage as deeply as we would like and we would not be able to bring a proposal for a review until time permits. Swift 3 is pretty ambitious as is, and losing focus can be a serious problem for a release.

Or perhaps a desire to have this sort of working group approach (similar to C++’s Technical Subcommittees), but not until 3.0 is released?

We can evaluate this again at some point, but I don't want to "fan out" to having multiple working groups until we've had more experience with this process through Swift 3. In part, this is because of my experiences with the ISO C++ process. Particularly now, where there are a significant number of active subgroups in the C++ committee, it is very hard to even keep track of all of the different pieces under discussion, much less establish and maintain a coherent vision going forward. And that’s in a fairly mature language (C++) where the community has had a longer time to get a feel for what the language is; it’s harder for a younger language like Swift where the philosophy isn’t as widely understood.

  - Doug

···

Sent from my iPhone
On Dec 12, 2015, at 1:36 PM, David Waite <david@alkaline-solutions.com <mailto:david@alkaline-solutions.com>> wrote:

-DW

On Dec 12, 2015, at 1:35 PM, Douglas Gregor via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Dec 11, 2015, at 10:48 PM, Kevin Ballard via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

I'd love to have first-class support for generators like this, but it's a lot of work. It's a lot easier to do this sort of thing in a scripting language like Python than it is to do in a language like Swift, because it requires reifying the stack into a data structure that can be passed around. And I suspect there's a lot of non-trivial questions that have to get answered before you can even propose an implementation for this.

For context, this sort of thing is something that people have been talking about doing in Rust for quite a while, and it keeps getting punted because of the amount of work and the unanswered questions about how it would actually be implemented.

So I'll give this a general +1, but I think it should also be deferred until after Swift 3 at the earliest.

Right. This is a major add-on feature that doesn’t fit in with the stated goals for Swift 3 (README of https://github.com/apple/swift-evolution\), so I think we should proactively defer it.

  - Doug

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

It'd be interesting to see if we can add some sort support for generator functions / yield functions to the language. As I believe we currently can only have a callback-style generator in which we have to manage the generator's state ourselves. It would be great to be able to use `yield` and not have to manage the state manually.

--Bouke

···

On 2015-12-13 22:32:06 +0000, Chris Lattner said:

On Dec 12, 2015, at 12:35 PM, Douglas Gregor via swift-evolution >> <swift-evolution at swift.org> wrote:

On Dec 11, 2015, at 10:48 PM, Kevin Ballard via swift-evolution >>> <swift-evolution at swift.org <mailto:swift-evolution at swift.org>> >>> wrote:

I'd love to have first-class support for generators like this, but it's a lot of work. It's a lot easier to do this sort of thing in a scripting language like Python than it is to do in a language like Swift, because it requires reifying the stack into a data structure that can be passed around. And I suspect there's a lot of non-trivial questions that have to get answered before you can even propose an implementation for this.

For context, this sort of thing is something that people have been talking about doing in Rust for quite a while, and it keeps getting punted because of the amount of work and the unanswered questions about how it would actually be implemented.

So I'll give this a general +1, but I think it should also be deferred until after Swift 3 at the earliest.

Right. This is a major add-on feature that doesn’t fit in with the stated goals for Swift 3 (README of https://github.com/apple/swift-evolution\), so I think we should proactively defer it.

I agree with both of you. I’m very interested in this, but it is clearly out of scope for Swift 3. It should also be considered alongside whatever async/concurrency approach we tackle (likely in swift 4).

-Chris

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20151213/4d8e5a36/attachment.html&gt;

Well, just returning "Generator<String>" doesn't distinguish this from a normal function that just returns a generator. Also, "Generator" is a protocol with an associated type, so you can't use it as a value type (yet?). Today you'd have to manually type-erase using the AnyGenerator wrapper. That also makes it harder to optimize, since clients won't know what kind of generator you're using. (Maybe that's a feature, though; it lets you change the implementation in the future. I guess it could make sense to have the interface show the "real" generator return type even if the source shows the element type.)

Jordan

···

On Dec 11, 2015, at 18:20 , Andrey Tarantsov <andrey@tarantsov.com> wrote:

Another possible syntax, inspired by "throws".

func foo() yield -> String

Well, again, yielding is an implementation detail. What's the advantage vs returning Generator<String>? I don't think that lying about the return type is useful for anyone.

I like this idea, as long as it is kept small in scope, only the
introduction of a yield implemented as something conforming to SequenceType.

You can get fairly concise syntax already with anySequence and AnyGenerator:

func fibonacci() -> AnySequence<Int> {

    return AnySequence<Int> { () -> AnyGenerator<Int> in

        var (i, j) = (0, 1)

        return anyGenerator {

            (i, j) = (j, i + j)

            return i

        }

    }

}

But state management is more involved.

To implement this I guess you'd have to look at each yield statement and
work out what state it can have, something like this:

func test() -> AnySequence<Int> {

    var j = 0

    for i in 1 ... 5 {

        j += i

        yield j

    }

    yield j

    return j+1

}

becomes:

func test() -> AnySequence<Int> {

    return AnySequence<Int> { () -> AnyGenerator<Int> in

        var state: YieldState_test = .Yield0(j: 0, i: 1)

        return anyGenerator { () -> Int? in

            switch state {

            case let .Yield0(j: j, i: i):

                let j_ = i + j, i_ = i+1

                state = (i_ <= 5) ? .Yield0(j: j_, i: i_) : .Yield1(j: j_)

                return j_

            case let .Yield1(j: j):

                state = .Return0(j)

                return j

            case let .Return0(r):

                state = .Complete

                return r+1

            case .Complete:

                return nil

            }

        }

    }

}

···

On Sat, Dec 12, 2015 at 10:20 PM, T.J. Usiyan via swift-evolution < swift-evolution@swift.org> wrote:

This feature (set) as sketched here by Joe is worthwhile, in my opinion.
The use of catch fits in and effect neatly captures this class of behaviors.
+1 for `effect`
---
TJ

On Sat, Dec 12, 2015 at 4:17 AM, Joe Groff via swift-evolution < > swift-evolution@swift.org> wrote:

On Dec 11, 2015, at 6:26 PM, Jordan Rose via swift-evolution < >> swift-evolution@swift.org> wrote:

Eh, I was trying to avoid grabbing another keyword, but I guess it's
context-sensitive anyway.

I've thought about this some. It might not have to be a keyword, if this
were a generalized language feature. Anything that interrupts control flow
and optionally resumes it later, such as 'throws', 'yields', and
potentially also 'async', could be implemented as instances of algebraic
effects. As a rough sketch of an idea, you could declare an effect and its
operations:

effect throws { @noreturn operation throw (ErrorType) -> () }
effect yields<T> { operation yield (T) -> () }
effect awaits { operation await<T> (Promise<T>) -> T }

and 'catch' could be generalized to let you handle any effect operations
that might be performed in the body of a block:

class Generator<T> {
var generator: () yields<T> -> ()
func next() -> T? {
do {
   generator()
return nil
} catch yield (let x) {
generator = currentContinuation
return x
}
}
}

See Eff (http://www.eff-lang.org) for an example of a language with this
already implemented.

-Joe

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Understood. C++ being more mature has more restrictions on what its subcommittees can really propose as changes to the language. This may make proposals harder, but likely helps them proceed in parallel. Swift likely has several more revolutionary releases ahead of it, and that makes it important to pool design into a few areas to try to increase their usefulness and composability.

-DW

···

On Dec 14, 2015, at 4:59 PM, Douglas Gregor <dgregor@apple.com> wrote:

We can evaluate this again at some point, but I don't want to "fan out" to having multiple working groups until we've had more experience with this process through Swift 3. In part, this is because of my experiences with the ISO C++ process. Particularly now, where there are a significant number of active subgroups in the C++ committee, it is very hard to even keep track of all of the different pieces under discussion, much less establish and maintain a coherent vision going forward. And that’s in a fairly mature language (C++) where the community has had a longer time to get a feel for what the language is; it’s harder for a younger language like Swift where the philosophy isn’t as widely understood.