Learning from SE-0025, a breeding group for Swift proposals

Evolution Discussion
1

Hello community,

I'm happy to see that SE-0169 got accepted and that we've patched the issues of SE-0025. But it's been a difficult process. And I can't stop asking myself if it could have been avoided. The crux of the problem is that source-compatibility is now becoming a very strong requirement. But at the same time, Swift is still missing some very big systems: reflection, property behaviours, a concurrency paradigm. How can we continue to push Swift boldly forward with very little leeway to correct our mistakes?

Then I listened to the latest episode of the excellent [Swift Unwrapped podcast](Spec · Level Up) where they talk about the access control "saga" and ask themselves the same questions as above. One interesting idea got my attention: JavaScript has a natural breeding ground for future language features with Babel. For those who don't know, it's a transcompiler that compiles bleeding-edge JavaScript into versions supported in browsers. Perhaps Swift could also benefit from a similar experimentation time for each new proposal.

Here's my rough idea:
The Swift compiler gains a new off-by-default `next` version triggerable with the `-swift-version next` flag.
All controversial proposals start their implementation in that version.
Once one of the poposals feels stable enough, it is brought into an official version.
Developers would be encouraged to try the `next` features while being warned that source compatibility on that version will *not* be garanteed.
As the vast majority of the Swift user base are still Apple platform developers, I think it would be important for the success of that strategy that the applications compiled with the `next` flag be accepted on the Apple stores or it will reduce the group of developers ready to play in this "breeding-group".

Any comments?

David.

2

I'm happy to see that SE-0169 got accepted and that we've patched the issues of SE-0025. But it's been a difficult process. And I can't stop asking myself if it could have been avoided. The crux of the problem is that source-compatibility is now becoming a very strong requirement. But at the same time, Swift is still missing some very big systems: reflection, property behaviours, a concurrency paradigm. How can we continue to push Swift boldly forward with very little leeway to correct our mistakes?

Then I listened to the latest episode of the excellent [Swift Unwrapped podcast](https://spec.fm/podcasts/swift-unwrapped\) where they talk about the access control "saga" and ask themselves the same questions as above. One interesting idea got my attention: JavaScript has a natural breeding ground for future language features with Babel. For those who don't know, it's a transcompiler that compiles bleeding-edge JavaScript into versions supported in browsers. Perhaps Swift could also benefit from a similar experimentation time for each new proposal.

There are a number of interesting examples in how other languages handle evolution.

For a language like ECMAScript, proposals follow a specific vetting process
https://tc39.github.io/process-document/

- proposals can proceed independent of one another
- a proposal is responsible for the full formal definition (think - if one of the processes was to update a Swift language specification)
- the process includes a feedback stage based on usage and implementation
- the process is not complete until there is full implementation in language implementations - the implementation challenges are feedback back into the process
- there isn’t a single review gate between something being a proposal and an accepted language feature
- the language is on a release schedule, and proposals are put into the language release schedule when complete. Language releases (e.g. ES2017) are not tied to a “MVP” set of proposed features
- because implementation is required as part of the process, ideally you do not have issues where the language spec is released with no implementations

I don’t know if the level of formalism here would work for Swift yet, to be honest. Swift still has known deficiencies that make sense to have be focus areas for a language revision. I suspect features within that revision schedule may also wind up being too interdependent. This division of labor also only makes sense when you have enough independent parties trying to drive evolution and implementation - Swift just might not be big enough yet.

Here's my rough idea:
The Swift compiler gains a new off-by-default `next` version triggerable with the `-swift-version next` flag.
All controversial proposals start their implementation in that version.
Once one of the poposals feels stable enough, it is brought into an official version.
Developers would be encouraged to try the `next` features while being warned that source compatibility on that version will *not* be garanteed.

Re: -swift-version next, care would have to be taken to make sure this doesn’t mean that multiple language features aren’t being developed within the same branch. There is also the risk that such features are *not* going to make the next release, or any release, due to implementation challenges or later user feedback. I’d recommend that this instead be an experimental version focused on periodic merging of beta features - there isn’t a requirement that such be a single tree, or be @ apple/swift, or to only contain features which have hit a certain process maturity stage.

I believe there are still challenges for users to experiment with such versions - in particular, I believe the majority of Apple platforms still don’t support building with anything other than an official release. I might expect a vendor running a store to require an official toolchain, but I believe even adhoc and local/simulator development is required to be on shipped toolchain releases.

-DW

···

On Apr 18, 2017, at 1:00 AM, David Hart via swift-evolution <swift-evolution@swift.org> wrote:

3

Just one more note here, in regards to SE-0025

Its important to realize that the swift evolution process isn’t a pure democracy, or even a democratic republic :-)

I personally suspect if '25 was truly controversial amongst the people who had proper votes (e.g. the core team), that it would not have been accepted. However, there was a desire to have such features. I think there may have been some pressure to have such a feature also included within the Swift 3 release, which was meant to be the last non-backward-source-compatible release.

I think SE-0169 and the limited choice within the Swift 4 (and all future Swift releases) exemplifies that desire by the core team - that model could be revised to allow splitting a type into extensions (in some contexts) without having to raise access control, and make the purpose of the different access control levels a bit clearer in the process.

IMHO, the current evolution process is not about letting the community vote, but to provide a larger pool of minds and eyes and recommendations to the core team.

-DW

···

On Apr 18, 2017, at 1:00 AM, David Hart via swift-evolution <swift-evolution@swift.org> wrote:

All controversial proposals start their implementation in that version.

4

All controversial proposals start their implementation in that version.

Just one more note here, in regards to SE-0025

Its important to realize that the swift evolution process isn’t a pure democracy, or even a democratic republic :-)

Of course :)

I personally suspect if '25 was truly controversial amongst the people who had proper votes (e.g. the core team), that it would not have been accepted. However, there was a desire to have such features. I think there may have been some pressure to have such a feature also included within the Swift 3 release, which was meant to be the last non-backward-source-compatible release.

I’ve had the impression that some proposals have been accepted in the past with some core team members being against. But even if all team members think a proposal is a good idea, they might still agree that its design or implementation could need to be refined after real-world use.

···

On 18 Apr 2017, at 10:12, David Waite <david@alkaline-solutions.com> wrote:

On Apr 18, 2017, at 1:00 AM, David Hart via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

I think SE-0169 and the limited choice within the Swift 4 (and all future Swift releases) exemplifies that desire by the core team - that model could be revised to allow splitting a type into extensions (in some contexts) without having to raise access control, and make the purpose of the different access control levels a bit clearer in the process.

IMHO, the current evolution process is not about letting the community vote, but to provide a larger pool of minds and eyes and recommendations to the core team.

-DW

5

very subjective {
I see a similarity with SE-0169:
It's fighting symptoms instead of causes.
}

(less subjective)
Afaics, even members of the core team have their quarrels with the evolution process, and I want to encourage everyone to talk openly about them.
Right now, we are at a turning point, with a big change in the structure of the team that happened recently and a big change of the tools that hopefully will happen soon (Discourse ftw! ;-), so I would appreciate if Chris and Ted (or the whole team) would sit together once again to write a retrospective — and ideally revise parts of the whole process based on the experience they have now.

6

I listened to the same podcast (which is generally great btw, even if it doesn’t get all the details right), but they miss an important fact: Swift *does* have an important beta cycle (typically started at WWDC) for major releases of the language. One very important thing with Swift 4 vs Swift 3 is that hopefully there is no feature work for Swift 4 after WWDC, meaning that the only changes are those that are responding to developer usage experience with the new stuff.

Also, it remains to be seen, but I strongly believe that we’ve ended up in a good place with access control. The process was painful, but worthwhile to reach a great result.

-Chris

···

On Apr 18, 2017, at 12:00 AM, David Hart via swift-evolution <swift-evolution@swift.org> wrote:

Hello community,

I'm happy to see that SE-0169 got accepted and that we've patched the issues of SE-0025. But it's been a difficult process. And I can't stop asking myself if it could have been avoided. The crux of the problem is that source-compatibility is now becoming a very strong requirement. But at the same time, Swift is still missing some very big systems: reflection, property behaviours, a concurrency paradigm. How can we continue to push Swift boldly forward with very little leeway to correct our mistakes?

Then I listened to the latest episode of the excellent [Swift Unwrapped podcast](https://spec.fm/podcasts/swift-unwrapped\) where they talk about the access control "saga" and ask themselves the same questions as above. One interesting idea got my attention: JavaScript has a natural breeding ground for future language features with Babel. For those who don't know, it's a transcompiler that compiles bleeding-edge JavaScript into versions supported in browsers. Perhaps Swift could also benefit from a similar experimentation time for each new proposal.

7

Hi David,

I think this is a well-intentioned suggestion, but I see several problems with this approach.

The first problem is one of expectations. Developers invest in using features because they want to write code to get their work done. Once someone starts shipping something that depends on a feature they get very grumpy when it gets taken away. It doesn’t really matter if the feature is “official” or not. Once enough people invest in using a feature it becomes “unofficially” part of the language. IMO, that doesn’t promote a model where changes to the language are well-considered.

The second problem is that within the compiler there is still vestigial remnants of earlier days in Swift when we were doing a lot of experimentation. Before Swift 1 was released we toyed around quite a bit with the syntax, the type checker, pretty much everything. We’d have a design meeting and then immediately someone would go implement something in the compiler. During those days such rapid experimentation was helpful to see how ideas would work out and to get a feel for the language we were crafting. But as time has gone by we’ve repeatedly found ourselves removing cruft from the compiler’s implementation and fixing bugs from technical debt of incompletely implemented features or features that didn’t work well together. Features added for the purpose of experimentation are almost always incomplete — it’s the very nature of experimentation. It is also very easy to convince yourself that a new language feature “works” when you try it out on a few examples.

No design process is going to be perfect. We made a lot design decisions before Swift was publicly announced that were made based on our intuition and experiences, but really those decisions weren’t evaluated until people started writing a lot of Swift code. We then have gone back — several times — and have revised core design decisions in the language based on learning from real-world usage of Swift. With the access control saga I think we saw another instance of that story: a lot of discussion happened in the Swift 3 timeframe, but some important factors/implications of the design being discussed that are now obvious in hindsight just weren’t apparent when the discussions were happening. I believe some number of these sagas are inevitable, and hopefully we learn from them to better inform our design decisions in the future.

I know the idea behind having a “-swift-version next” is to allow people to invest in experimental ideas to provide real feedback, but I don’t think this encourages good language design and many developers would be fearful of using such experimental features. Those that fall in love with an experimental feature would likely fervently defend their existence, even if they end up being a bad idea. It’s also not clear by what criteria we would take experimental features into the compiler in the first place, and adding more cruft into the compiler could be very detrimental to its implementation quality.

As Chris said, Xcode betas really are a good time for people to provide feedback about features. With Swift 3 some of that feedback was delayed because of the upheaval caused by the Grand API Renaming and the migration work projects needed to do to go from Swift 2 to Swift 3. Going forward the intent is that the evolution of Swift provides a much smoother, continuous experience for users, and allows them to try out new features of the language more easily and thus possible provide feedback sooner.

Ted

···

On Apr 18, 2017, at 12:00 AM, David Hart via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Here's my rough idea:
The Swift compiler gains a new off-by-default `next` version triggerable with the `-swift-version next` flag.
All controversial proposals start their implementation in that version.
Once one of the poposals feels stable enough, it is brought into an official version.
Developers would be encouraged to try the `next` features while being warned that source compatibility on that version will *not* be garanteed.
As the vast majority of the Swift user base are still Apple platform developers, I think it would be important for the success of that strategy that the applications compiled with the `next` flag be accepted on the Apple stores or it will reduce the group of developers ready to play in this "breeding-group".

8

Hello community,

I'm happy to see that SE-0169 got accepted and that we've patched the issues of SE-0025. But it's been a difficult process. And I can't stop asking myself if it could have been avoided. The crux of the problem is that source-compatibility is now becoming a very strong requirement. But at the same time, Swift is still missing some very big systems: reflection, property behaviours, a concurrency paradigm. How can we continue to push Swift boldly forward with very little leeway to correct our mistakes?

Then I listened to the latest episode of the excellent [Swift Unwrapped podcast](Spec · Level Up) where they talk about the access control "saga" and ask themselves the same questions as above. One interesting idea got my attention: JavaScript has a natural breeding ground for future language features with Babel. For those who don't know, it's a transcompiler that compiles bleeding-edge JavaScript into versions supported in browsers. Perhaps Swift could also benefit from a similar experimentation time for each new proposal.

I listened to the same podcast (which is generally great btw, even if it doesn’t get all the details right), but they miss an important fact: Swift *does* have an important beta cycle (typically started at WWDC) for major releases of the language. One very important thing with Swift 4 vs Swift 3 is that hopefully there is no feature work for Swift 4 after WWDC, meaning that the only changes are those that are responding to developer usage experience with the new stuff.

That's true. In that case, I think the problem with SE-0025 was that it was implemented very late in the Beta cycle (by no fault of the dev team, you had so much on your plate) so we had very little time to experiment with the feature.

Hopefully, with the tightening of scope in SE proposals, most can get implemented early in the beta cycle.

Also, it remains to be seen, but I strongly believe that we’ve ended up in a good place with access control. The process was painful, but worthwhile to reach a great result.

I agree :) I just want to avoid painting ourselves in a backwards-compatibility-corner with another major feature.

···

On 19 Apr 2017, at 06:51, Chris Lattner <clattner@nondot.org> wrote:

On Apr 18, 2017, at 12:00 AM, David Hart via swift-evolution <swift-evolution@swift.org> wrote:

-Chris

9

Once someone starts shipping something that depends on a feature they get very grumpy when it gets taken away.

Well, it happened before, and people's life went on without tuple splat and currying… ;-)
I don't think an open Beta would add that much value on its own — but imho the aspect of temporary acceptance could improve the process:
Instead of being implemented right away (in theory ;-), changes that really alter the shape of Swift could be delayed for a longer period to allow other proposals to be build on them.
Such a proposal would define the general direction, and it should be the goal to incorporate it in a future release — but if it turns out that the original path has issues that are revealed by constructs built on top of it, the initial idea could still be improved without causing churn.

As a sidenote, originally I merely thought SE-0025 is nonsense, and I just didn't oppose strongly because, after all, replacing "private" with "fileprivate" isn't that terrible. But during the discussions for Swift 4, I changed my mind, and now I think that with some follow-ups, the concept of scope-private could have become really useful...