Large Proposal: Non-Standard Libraries

[davedelong]

Hi Swift-Evolution,

The Standard Library's goal is to be small and targeted. However, many aspects of Apple-provided frameworks need or offer opportunities for improvement or wholesale replacement. These enhancements lie beyond the scope of the Standard Library.

To address this, we'd like to propose the idea of a "Non-Standard Library"; this would be a library that ships with a regular installation of Swift, but is not imported into .swift files by the compiler, unless explicitly requested by the developer.

We are proposing a well-organized effort to parallel the Standard Library without putting additional implementation responsibilities onto the core team. This effort would mitigate what we see as platform-independent requirements that provide native Swift implementations that aren't burdened by Apple history.

Mission Statement

We propose to create an open-sourced "Non-Standard Library" effort that coexists with, coordinates with, and is blessed by the open source Swift development community. The "Non-Standard Library" will offer a well-curated, high-value collection of routines, types, and documentation to support Swift development on all platforms.

Goals

The main goals of this effort would be:
Alleviate pressure on the Core Team while providing the developer community with functionality that normally falls under Apple's internal development umbrella.
Deliver authoritative, reliable, and regularly updated libraries avoiding issues faced by third-party dependencies.
Provide oversight, organization, and full community involvement to ensure its components are worthy, maintained, and passing a bar of need and excellence.
Suggested Libraries

There are several areas we think that are ripe for improvement and implementation as a non-standard library, such as (but not limited to):
A BigNum library
A full-featured Random library
A simplified date/time library
A library for manipulating paths (that is not based on URL or String)
An expanded Swift-native Geometry library and so forth.
The scope and extent of the sublibraries would be proposed and debated on a parallel Non-Standard Library Evolution development list.

Coordination

This effort would be fully coordinated with the Swift development team, respecting the team's existing commitments and responsibilities. We would request an Apple body to act as an official coordinator, enabling both oversight of this effort and to expose Apple-sourced suggestions to the Non-Standard community for action.

Next Steps

To proceed, we need a general community consensus that this effort is worth the significant overhead it would involve.

We must then:
Select a project lead, who appoints technical leaders from the community.
Recruit a core team, a small group responsible for strategic direction, pulled from experienced members well versed in contributing to Swift-dev.
Establish a Non-Standard Swift Evolution process, based on the one that is currently followed by Swift Evolution. Following SE practices will guarantee a consistent and excellent language experience for those developers including the Non-Standard library into their projects.
Build a Non-Standard Swift Evolution repository home.
Adopt a code license, based on Swift's current licensing.
Define the community working group rules and code of conduct.
Establish a mailing list and/or forum.
Begin the selection of target sublibraries and populating them by recruiting committers and contributors.
Alternative Names

Suggested names for this effort include the following.
Extended-Standard Library
Community-Standard Library
Non-Standard Library
We are not married to any of these names and welcome alternative suggestions.

Measures of Success

A successful Non-Standard Library will provide a stable and incrementally grown ecology of Swift frameworks that developers can depend upon with a minimum of turbulence and regret cycles. For that reason, the most successful content will be core functionality, with significant field testing prior to adoption. We recommend that NSSE follow a model of provisionary adoption and refinement before deployment to ensure any adopting code base will not be affected by later module changes.

Thanks,

Dave DeLong and Erica Sadun

[taylorswift]

This is something I have been pushing for for some time now and I am glad
to see some more force behind it. There are multiple prior threads about
this topic you should probably familiarize yourselves with for background:

https://lists.swift.org/pipermail/swift-evolution/
Week-of-Mon-20160125/007815.html

https://lists.swift.org/pipermail/swift-evolution/
Week-of-Mon-20170731/038401.html

Since it looks like cross-module inlining/specialization is slated for
Swift 5 I believe now is a good time to get such an effort started as this
was really the only *technical* barrier preventing such libraries from
being written in the past.

Hi Swift-Evolution,

The Standard Library's goal is to be small and targeted. However, many
aspects of Apple-provided frameworks need or offer opportunities for
improvement or wholesale replacement. These enhancements lie beyond the
scope of the Standard Library.

To address this, we'd like to propose the idea of a "Non-Standard
Library"; this would be a library that ships with a regular installation of
Swift, but is not imported into .swift files by the compiler, unless
explicitly requested by the developer.

We are proposing a well-organized effort to parallel the Standard Library
without putting additional implementation responsibilities onto the core
team. This effort would mitigate what we see as platform-independent
requirements that provide native Swift implementations that aren't
burdened by Apple history.

*Mission Statement*

We propose to create an open-sourced "Non-Standard Library" effort that
coexists with, coordinates with, and is blessed by the open source Swift
development community. The "Non-Standard Library" will offer a
well-curated, high-value collection of routines, types, and documentation
to support Swift development on all platforms.

*Goals*

The main goals of this effort would be:

   - Alleviate pressure on the Core Team while providing the developer
   community with functionality that normally falls under Apple's internal
   development umbrella.
   - Deliver authoritative, reliable, and regularly updated libraries
   avoiding issues faced by third-party dependencies.
   - Provide oversight, organization, and full community involvement to
   ensure its components are worthy, maintained, and passing a bar of need and
   excellence.

*Suggested Libraries*

There are several areas we think that are ripe for improvement and
implementation as a non-standard library, such as (but not limited to):

   - A BigNum library
   - A full-featured Random library
   - A simplified date/time library
   - A library for manipulating paths (that is not based on URL or String)
   - An expanded Swift-native Geometry library and so forth.

Random is currently being discussed for inclusion into the stdlib. I’m

currently developing <https://github.com/kelvin13/url&gt; a modern URI type to
replace the Foundation type.

The scope and extent of the sublibraries would be proposed and debated on
a parallel Non-Standard Library Evolution development list.

*Coordination*

This effort would be fully coordinated with the Swift development team,
respecting the team's existing commitments and responsibilities. We would
request an Apple body to act as an official coordinator, enabling both
oversight of this effort and to expose Apple-sourced suggestions to
the Non-Standard community for action.

I love Apple but is Apple oversight really going to be beneficial enough to
warrant the additional bureaucratic overhead?

*Next Steps*

To proceed, we need a general community consensus that this effort is
worth the significant overhead it would involve.

We must then:

   - Select a project lead, who appoints technical leaders from the
   community.

Yes

   - Recruit a core team, a small group responsible for strategic
   direction, pulled from experienced members well versed in contributing to
   Swift-dev.

Yes

   - Establish a Non-Standard Swift Evolution process, based on the one
   that is currently followed by Swift Evolution. Following SE practices will
   guarantee a consistent and excellent language experience for those
   developers including the Non-Standard library into their projects.

I disagree. Swift evolution is incredibly brittle and bureaucratic, it

only works for the stdlib because it is relatively small and most of it has
already been solidified. Any non-standard library initiative would easily
overwhelm the bandwidth of a mailing list. Project leads should have
considerably more autonomy than for the stdlib.

   - Build a Non-Standard Swift Evolution repository home.

Yes, and might I add this is a lot more important than most people seem to

acknowledge.

   - Adopt a code license, based on Swift's current licensing.

Yes

   - Define the community working group rules and code of conduct.

Yes. Since these libraries are meant to replace large parts of Foundation,

we should probably ban Foundation imports.

···

On Tue, Nov 7, 2017 at 11:54 AM, Dave DeLong via swift-evolution < swift-evolution@swift.org> wrote:

   - Establish a mailing list and/or forum.

Sure

   - Begin the selection of target sublibraries and populating them by
   recruiting committers and contributors.

Yep

*Alternative Names*

Suggested names for this effort include the following.

   - Extended-Standard Library
   - Community-Standard Library
   - Non-Standard Library

We are not married to any of these names and welcome alternative
suggestions.

*Measures of Success*

A successful Non-Standard Library will provide a stable and incrementally
grown ecology of Swift frameworks that developers can depend upon with a
minimum of turbulence and regret cycles. For that reason, the most
successful content will be core functionality, with significant field
testing prior to adoption. We recommend that NSSE follow a model of
provisionary adoption and refinement before deployment to ensure any
adopting code base will not be affected by later module changes.

Thanks,

Dave DeLong and Erica Sadun

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[tkremenek (Ted Kremenek)]

Hi Dave,

Thanks for bringing up this topic. This has been kicked around a little, and we’re still exploring different models on how to extend Swift.

The server APIs work group is one operational model for the community to build out a new set of core libraries. That work group was formed out of the observation that there were different implementations of the same thing being created by different Swift on server efforts, and that those different efforts should pool those resources together and standardize on some fundamentals.

That analogy could work here. However, it also has possible major downsides. It can be heavyweight, and also artificially raise the importance of certain people’s library efforts over others by creating a formal work group whose efforts are expected to eventually be incorporated into the core Swift distribution. It would also force a discussion up front of what even makes sense to incorporate into the core Swift distribution, which might be artificially constraining the exploration of the set of libraries to implement.

I need to think about your idea more, but my first reaction is that my preferred route is that the community builds these libraries, ideally using Swift Packages, and through trial and use we evaluate those libraries and then decide if they should be incorporated as part of the core distribution. There’s many factors involved in deciding if they can be incorporated into the core distribution, but all of those could be informed by actually building libraries and see them getting used.

We could also figure out a way to possibly highlight these efforts to the Swift community, maybe on swift-evolution or other means — but all with the expectation that these libraries are not *necessarily* going to be standardized as part of the core swift distribution. I don’t think that’s all that bad either; not every library will make sense to incorporate into the core Swift distribution (e.g., they are highly domain specific) but still supporting their development would be beneficial to the community.

Note that any change going into the Swift core distribution inevitably involves swift-evolution and the core team. Changes going into the core Swift distribution must meet a very high bar as far as implementation and design, the confidence we have into committing to specific APIs (especially since we’ll have ABI stability in Swift 5), and other factors. Thus this will not really alleviate much burden on the Core team or on the community — one of the core goals of your proposal. Further, incorporating all those concerns up front when building libraries might be counterproductive.

FWIW, Ben Cohen and I have been talking about possibly using Swift packages as a way to seed out experimental ideas for extensions to the Standard Library. This would allow ideas to be trialed by real usage (a complaint I’ve seen about some changes we’ve made to Swift in the past). Users could build things on top of those libraries, knowing they are available as packages, and if an API “graduates” to being part of the Standard Library the user can then depend upon it being available there. If it never graduates, however, the package remains around.

One thing that resonates me in what you propose is about having a “well-organized effort” whose aim is to make these libraries feel cohesive and implemented well. However, given that many of these naturally fall somewhere in the spectrum of responsibilities within the Standard Library and Foundation, I think it is self-misleading to think that the Core Team or others would not be involved in these efforts if the intention is to possibly incorporate these one day into the core Swift distribution. There also may be other ways to achieve that level of cohesion in API design, such as through community discussion (possibly via the Swift.org <http://swift.org/&gt; mailing lists/forums). This discussion would not necessarily be the same as the path to “ratification” of a library into core Swift, but a step that could benefit many library authors (including considering ideas one day incorporated into the core swift). With such a mechanism many library authors could benefit even if they do not intend to have the library as part of the core distribution.

My apologies that these are all hand-wavy ideas, but I’m trying to paint a possible alternative way to achieve your goal of more library evolution for Swift without having such a formal work group that may be too heavy weight or too narrowly focused.

Ted

···

On Nov 7, 2017, at 9:54 AM, Dave DeLong via swift-evolution <swift-evolution@swift.org> wrote:

Hi Swift-Evolution,

The Standard Library's goal is to be small and targeted. However, many aspects of Apple-provided frameworks need or offer opportunities for improvement or wholesale replacement. These enhancements lie beyond the scope of the Standard Library.

To address this, we'd like to propose the idea of a "Non-Standard Library"; this would be a library that ships with a regular installation of Swift, but is not imported into .swift files by the compiler, unless explicitly requested by the developer.

We are proposing a well-organized effort to parallel the Standard Library without putting additional implementation responsibilities onto the core team. This effort would mitigate what we see as platform-independent requirements that provide native Swift implementations that aren't burdened by Apple history.

Mission Statement

We propose to create an open-sourced "Non-Standard Library" effort that coexists with, coordinates with, and is blessed by the open source Swift development community. The "Non-Standard Library" will offer a well-curated, high-value collection of routines, types, and documentation to support Swift development on all platforms.

Goals

The main goals of this effort would be:
Alleviate pressure on the Core Team while providing the developer community with functionality that normally falls under Apple's internal development umbrella.
Deliver authoritative, reliable, and regularly updated libraries avoiding issues faced by third-party dependencies.
Provide oversight, organization, and full community involvement to ensure its components are worthy, maintained, and passing a bar of need and excellence.
Suggested Libraries

There are several areas we think that are ripe for improvement and implementation as a non-standard library, such as (but not limited to):
A BigNum library
A full-featured Random library
A simplified date/time library
A library for manipulating paths (that is not based on URL or String)
An expanded Swift-native Geometry library and so forth.
The scope and extent of the sublibraries would be proposed and debated on a parallel Non-Standard Library Evolution development list.

Coordination

This effort would be fully coordinated with the Swift development team, respecting the team's existing commitments and responsibilities. We would request an Apple body to act as an official coordinator, enabling both oversight of this effort and to expose Apple-sourced suggestions to the Non-Standard community for action.

Next Steps

To proceed, we need a general community consensus that this effort is worth the significant overhead it would involve.

We must then:
Select a project lead, who appoints technical leaders from the community.
Recruit a core team, a small group responsible for strategic direction, pulled from experienced members well versed in contributing to Swift-dev.
Establish a Non-Standard Swift Evolution process, based on the one that is currently followed by Swift Evolution. Following SE practices will guarantee a consistent and excellent language experience for those developers including the Non-Standard library into their projects.
Build a Non-Standard Swift Evolution repository home.
Adopt a code license, based on Swift's current licensing.
Define the community working group rules and code of conduct.
Establish a mailing list and/or forum.
Begin the selection of target sublibraries and populating them by recruiting committers and contributors.
Alternative Names

Suggested names for this effort include the following.
Extended-Standard Library
Community-Standard Library
Non-Standard Library
We are not married to any of these names and welcome alternative suggestions.

Measures of Success

A successful Non-Standard Library will provide a stable and incrementally grown ecology of Swift frameworks that developers can depend upon with a minimum of turbulence and regret cycles. For that reason, the most successful content will be core functionality, with significant field testing prior to adoption. We recommend that NSSE follow a model of provisionary adoption and refinement before deployment to ensure any adopting code base will not be affected by later module changes.

Thanks,

Dave DeLong and Erica Sadun
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[Tino]

Great news — I think this will not only push Swift forward, but also empower more people to contribute to the ecosystem:
With the updated rules for evolution proposals, the bar to actively contribute to the compiler has been raised significantly; non-standard libraries on the other hand can be improved without knowing the details of llvm (you wouldn’t even need C++ ;-).

I suggest to focus on trivial data structures first, so that more advanced (and really non-standard libs ;-) can be build on a common basis (my favorite example here is the concept of a quaternion — iOS/Mac developers can encounter several incarnations of them without looking at third-party frameworks).

[Chris_Lattner]

Hi Dave,

As others have pointed out, we do already have a model for this sort of thing: the swift server working group.

That said, there is another analogy which gets closer to what you’re asking for: the Boost community for C++. Boost was formed because the C++ committee was too bogged down an wasn’t receptive to major library changes (at one point in time). Boost has effectively parallel leadership from the C++ committee (though individuals are involved in both organizations of course). This allows Boost to move faster, ship code, and get experience with it.

One of the specifically nice things about Boost is that they (at least originally) focused on building out ideas, getting experience with them, and then bringing the libraries back to the standard. The libraries occasionally undergo significant change when they are standardized, but the usage experience is unmatchable, particularly for very large and complex APIs.

In the context of Swift, I think this sort of model could be very interesting, because there are really several different independent things going on: for a type like BigNum (for example) there are all the details of the implementation and design on the one hand, but then there is also the question of WHICH library it should ship with (Foundation or Swift or something else). That second decision is much easier to make after the community has converged on a specific design.

In any case, I think it would be a bad move for the official Swift distributions to ship code that hasn’t been through the evolution process. The idea of the Server working group is to delegate detailed design and iteration to a team of experts, but then have them bring back the API to evolution when the iteration is done and it is time to “standardize” it. I think that this is a good model.

-Chris

···

On Nov 7, 2017, at 5:54 PM, Dave DeLong via swift-evolution <swift-evolution@swift.org> wrote:

Hi Swift-Evolution,

The Standard Library's goal is to be small and targeted. However, many aspects of Apple-provided frameworks need or offer opportunities for improvement or wholesale replacement. These enhancements lie beyond the scope of the Standard Library.

To address this, we'd like to propose the idea of a "Non-Standard Library"; this would be a library that ships with a regular installation of Swift, but is not imported into .swift files by the compiler, unless explicitly requested by the developer.

We are proposing a well-organized effort to parallel the Standard Library without putting additional implementation responsibilities onto the core team. This effort would mitigate what we see as platform-independent requirements that provide native Swift implementations that aren't burdened by Apple history.

[Benjamin_Spratling]

Sound great!

Last week I started working on a pure-swift graphics library, one goal being fast server-side graphics manipulations, and already have .png decode/encode, and quadratic bezier curve stroking implemented, slowly, and poorly. I’m working on TrueType fonts right now, and intend to open it to other developers once the basic architecture is refined. I intend to add SVG and JPEG support as well. It seems like if there is a “swift-community blessed geometry library”, this would be a natural extension of that, or at least depend on it.

Much of this depends on compression support, which I’ve already published a first cut of at GitHub - benspratling4/SwiftFoundationCompression: Foundation-oriented Swift wrapper for zlib . The goal there was a more “foundation” idea of compression than competing libraries, but it suffers from a lack of streaming support.

I would absolutely love a “BigNum” library. Several encryption packages out there would benefit from it as well. It seems arithmetic compression libraries would benefit, too. Other useful Numeric types would include rational fractions, fixed-point, and short floats (Float16 is what Apple now thinks the future of images is, right?). I’m happy to contribute; I end up writing those in a new language ever few years anyway.

-Ben Spratling

···

On Nov 7, 2017, at 12:54 PM, Dave DeLong via swift-evolution <swift-evolution@swift.org> wrote:

Hi Swift-Evolution,
...

We propose to create an open-sourced "Non-Standard Library" effort that coexists with, coordinates with, and is blessed by the open source Swift development community. The "Non-Standard Library" will offer a well-curated, high-value collection of routines, types, and documentation to support Swift development on all platforms.
…

Suggested Libraries

There are several areas we think that are ripe for improvement and implementation as a non-standard library, such as (but not limited to):
A BigNum library
A full-featured Random library
A simplified date/time library
A library for manipulating paths (that is not based on URL or String)
An expanded Swift-native Geometry library and so forth.
The scope and extent of the sublibraries would be proposed and debated on a parallel Non-Standard Library Evolution development list.

[David_Sweeris]

FWIW, I don't think it much matters if they get rolled into the "core" Swift distribution or library... I think all that's really necessary (well, to the extent that any of this is) is just to have an easy way to search for & install packages. Even just having the commonly used ones bundled into a "not quite standard library" that's a separate download would be icing on the cake, IMHO, because it makes it easy to find them and know that with them being widely used, they're probably at least reasonably fit for purpose.

- Dave Sweeris

···

On Nov 7, 2017, at 1:58 PM, Ted Kremenek via swift-evolution <swift-evolution@swift.org> wrote:

Hi Dave,

Thanks for bringing up this topic. This has been kicked around a little, and we’re still exploring different models on how to extend Swift.

The server APIs work group is one operational model for the community to build out a new set of core libraries. That work group was formed out of the observation that there were different implementations of the same thing being created by different Swift on server efforts, and that those different efforts should pool those resources together and standardize on some fundamentals.

That analogy could work here. However, it also has possible major downsides. It can be heavyweight, and also artificially raise the importance of certain people’s library efforts over others by creating a formal work group whose efforts are expected to eventually be incorporated into the core Swift distribution. It would also force a discussion up front of what even makes sense to incorporate into the core Swift distribution, which might be artificially constraining the exploration of the set of libraries to implement.

I need to think about your idea more, but my first reaction is that my preferred route is that the community builds these libraries, ideally using Swift Packages, and through trial and use we evaluate those libraries and then decide if they should be incorporated as part of the core distribution. There’s many factors involved in deciding if they can be incorporated into the core distribution, but all of those could be informed by actually building libraries and see them getting used.

We could also figure out a way to possibly highlight these efforts to the Swift community, maybe on swift-evolution or other means — but all with the expectation that these libraries are not *necessarily* going to be standardized as part of the core swift distribution. I don’t think that’s all that bad either; not every library will make sense to incorporate into the core Swift distribution (e.g., they are highly domain specific) but still supporting their development would be beneficial to the community.

Note that any change going into the Swift core distribution inevitably involves swift-evolution and the core team. Changes going into the core Swift distribution must meet a very high bar as far as implementation and design, the confidence we have into committing to specific APIs (especially since we’ll have ABI stability in Swift 5), and other factors. Thus this will not really alleviate much burden on the Core team or on the community — one of the core goals of your proposal. Further, incorporating all those concerns up front when building libraries might be counterproductive.

FWIW, Ben Cohen and I have been talking about possibly using Swift packages as a way to seed out experimental ideas for extensions to the Standard Library. This would allow ideas to be trialed by real usage (a complaint I’ve seen about some changes we’ve made to Swift in the past). Users could build things on top of those libraries, knowing they are available as packages, and if an API “graduates” to being part of the Standard Library the user can then depend upon it being available there. If it never graduates, however, the package remains around.

One thing that resonates me in what you propose is about having a “well-organized effort” whose aim is to make these libraries feel cohesive and implemented well. However, given that many of these naturally fall somewhere in the spectrum of responsibilities within the Standard Library and Foundation, I think it is self-misleading to think that the Core Team or others would not be involved in these efforts if the intention is to possibly incorporate these one day into the core Swift distribution. There also may be other ways to achieve that level of cohesion in API design, such as through community discussion (possibly via the Swift.org <http://swift.org/&gt; mailing lists/forums). This discussion would not necessarily be the same as the path to “ratification” of a library into core Swift, but a step that could benefit many library authors (including considering ideas one day incorporated into the core swift). With such a mechanism many library authors could benefit even if they do not intend to have the library as part of the core distribution.

My apologies that these are all hand-wavy ideas, but I’m trying to paint a possible alternative way to achieve your goal of more library evolution for Swift without having such a formal work group that may be too heavy weight or too narrowly focused.

[Jon_Hull]

What I really want to see is a Code Store in Xcode (kind of like an app store for Swift packages). Basically, you would hit “Add Package” in the sidebar/menu and a sheet pops up where you can search for and view various packages. You then hit an “Add” button in the sheet, and it would add the package to your project. To remove a package you would select it and hit delete.

Selecting a package in the sidebar would let you set some settings (e.g. select a specific version), and view documentation. Xcode would manage the files for you behind the scenes.

I know that is out of scope for Evolution, but I just wanted to mention it, since it might help with the visibility issue.

Thanks,
Jon

···

On Nov 7, 2017, at 1:58 PM, Ted Kremenek via swift-evolution <swift-evolution@swift.org> wrote:

We could also figure out a way to possibly highlight these efforts to the Swift community, maybe on swift-evolution or other means — but all with the expectation that these libraries are not *necessarily* going to be standardized as part of the core swift distribution. I don’t think that’s all that bad either; not every library will make sense to incorporate into the core Swift distribution (e.g., they are highly domain specific) but still supporting their development would be beneficial to the community.

[Alejandro_Martinez]

I’m with Ted on this one.
I would also love to see the Swift ecosystem grow but I think that it has to happen with SPM. With improvements on SPM (as discussed in other threads) and having a proper index (imho Cocoapods webpage is the best one out there, with stats about docs, unit testing, downloads...) that makes finding libraries easier.
But also a discussion forum where great libs can be shared and highlighted to the rest of the community and big community efforts coordinated. If you look at Rust is a great example, their forums attract the whole community (sadly not something the mailing list does) and big and important projects have come up from the community that have made a huge impact. With that in place and some libs out there maturing I’m sure that in the future the conversation to include battle tested code into an oficial distribution would be easier.

···

Sent from my iPad

On 7 Nov 2017, at 21:58, Ted Kremenek via swift-evolution <swift-evolution@swift.org> wrote:

Hi Dave,

Thanks for bringing up this topic. This has been kicked around a little, and we’re still exploring different models on how to extend Swift.

The server APIs work group is one operational model for the community to build out a new set of core libraries. That work group was formed out of the observation that there were different implementations of the same thing being created by different Swift on server efforts, and that those different efforts should pool those resources together and standardize on some fundamentals.

That analogy could work here. However, it also has possible major downsides. It can be heavyweight, and also artificially raise the importance of certain people’s library efforts over others by creating a formal work group whose efforts are expected to eventually be incorporated into the core Swift distribution. It would also force a discussion up front of what even makes sense to incorporate into the core Swift distribution, which might be artificially constraining the exploration of the set of libraries to implement.

I need to think about your idea more, but my first reaction is that my preferred route is that the community builds these libraries, ideally using Swift Packages, and through trial and use we evaluate those libraries and then decide if they should be incorporated as part of the core distribution. There’s many factors involved in deciding if they can be incorporated into the core distribution, but all of those could be informed by actually building libraries and see them getting used.

We could also figure out a way to possibly highlight these efforts to the Swift community, maybe on swift-evolution or other means — but all with the expectation that these libraries are not *necessarily* going to be standardized as part of the core swift distribution. I don’t think that’s all that bad either; not every library will make sense to incorporate into the core Swift distribution (e.g., they are highly domain specific) but still supporting their development would be beneficial to the community.

Note that any change going into the Swift core distribution inevitably involves swift-evolution and the core team. Changes going into the core Swift distribution must meet a very high bar as far as implementation and design, the confidence we have into committing to specific APIs (especially since we’ll have ABI stability in Swift 5), and other factors. Thus this will not really alleviate much burden on the Core team or on the community — one of the core goals of your proposal. Further, incorporating all those concerns up front when building libraries might be counterproductive.

FWIW, Ben Cohen and I have been talking about possibly using Swift packages as a way to seed out experimental ideas for extensions to the Standard Library. This would allow ideas to be trialed by real usage (a complaint I’ve seen about some changes we’ve made to Swift in the past). Users could build things on top of those libraries, knowing they are available as packages, and if an API “graduates” to being part of the Standard Library the user can then depend upon it being available there. If it never graduates, however, the package remains around.

One thing that resonates me in what you propose is about having a “well-organized effort” whose aim is to make these libraries feel cohesive and implemented well. However, given that many of these naturally fall somewhere in the spectrum of responsibilities within the Standard Library and Foundation, I think it is self-misleading to think that the Core Team or others would not be involved in these efforts if the intention is to possibly incorporate these one day into the core Swift distribution. There also may be other ways to achieve that level of cohesion in API design, such as through community discussion (possibly via the Swift.org mailing lists/forums). This discussion would not necessarily be the same as the path to “ratification” of a library into core Swift, but a step that could benefit many library authors (including considering ideas one day incorporated into the core swift). With such a mechanism many library authors could benefit even if they do not intend to have the library as part of the core distribution.

My apologies that these are all hand-wavy ideas, but I’m trying to paint a possible alternative way to achieve your goal of more library evolution for Swift without having such a formal work group that may be too heavy weight or too narrowly focused.

Ted

On Nov 7, 2017, at 9:54 AM, Dave DeLong via swift-evolution <swift-evolution@swift.org> wrote:

Hi Swift-Evolution,

The Standard Library's goal is to be small and targeted. However, many aspects of Apple-provided frameworks need or offer opportunities for improvement or wholesale replacement. These enhancements lie beyond the scope of the Standard Library.

To address this, we'd like to propose the idea of a "Non-Standard Library"; this would be a library that ships with a regular installation of Swift, but is not imported into .swift files by the compiler, unless explicitly requested by the developer.

We are proposing a well-organized effort to parallel the Standard Library without putting additional implementation responsibilities onto the core team. This effort would mitigate what we see as platform-independent requirements that provide native Swift implementations that aren't burdened by Apple history.

Mission Statement

We propose to create an open-sourced "Non-Standard Library" effort that coexists with, coordinates with, and is blessed by the open source Swift development community. The "Non-Standard Library" will offer a well-curated, high-value collection of routines, types, and documentation to support Swift development on all platforms.

Goals

The main goals of this effort would be:
Alleviate pressure on the Core Team while providing the developer community with functionality that normally falls under Apple's internal development umbrella.
Deliver authoritative, reliable, and regularly updated libraries avoiding issues faced by third-party dependencies.
Provide oversight, organization, and full community involvement to ensure its components are worthy, maintained, and passing a bar of need and excellence.
Suggested Libraries

There are several areas we think that are ripe for improvement and implementation as a non-standard library, such as (but not limited to):
A BigNum library
A full-featured Random library
A simplified date/time library
A library for manipulating paths (that is not based on URL or String)
An expanded Swift-native Geometry library and so forth.
The scope and extent of the sublibraries would be proposed and debated on a parallel Non-Standard Library Evolution development list.

Coordination

This effort would be fully coordinated with the Swift development team, respecting the team's existing commitments and responsibilities. We would request an Apple body to act as an official coordinator, enabling both oversight of this effort and to expose Apple-sourced suggestions to the Non-Standard community for action.

Next Steps

To proceed, we need a general community consensus that this effort is worth the significant overhead it would involve.

We must then:
Select a project lead, who appoints technical leaders from the community.
Recruit a core team, a small group responsible for strategic direction, pulled from experienced members well versed in contributing to Swift-dev.
Establish a Non-Standard Swift Evolution process, based on the one that is currently followed by Swift Evolution. Following SE practices will guarantee a consistent and excellent language experience for those developers including the Non-Standard library into their projects.
Build a Non-Standard Swift Evolution repository home.
Adopt a code license, based on Swift's current licensing.
Define the community working group rules and code of conduct.
Establish a mailing list and/or forum.
Begin the selection of target sublibraries and populating them by recruiting committers and contributors.
Alternative Names

Suggested names for this effort include the following.
Extended-Standard Library
Community-Standard Library
Non-Standard Library
We are not married to any of these names and welcome alternative suggestions.

Measures of Success

A successful Non-Standard Library will provide a stable and incrementally grown ecology of Swift frameworks that developers can depend upon with a minimum of turbulence and regret cycles. For that reason, the most successful content will be core functionality, with significant field testing prior to adoption. We recommend that NSSE follow a model of provisionary adoption and refinement before deployment to ensure any adopting code base will not be affected by later module changes.

Thanks,

Dave DeLong and Erica Sadun
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[taylorswift]

>
> Hi Swift-Evolution,
>
> The Standard Library's goal is to be small and targeted. However, many
aspects of Apple-provided frameworks need or offer opportunities for
improvement or wholesale replacement. These enhancements lie beyond the
scope of the Standard Library.
>
> To address this, we'd like to propose the idea of a "Non-Standard
Library"; this would be a library that ships with a regular installation of
Swift, but is not imported into .swift files by the compiler, unless
explicitly requested by the developer.
>
> We are proposing a well-organized effort to parallel the Standard
Library without putting additional implementation responsibilities onto the
core team. This effort would mitigate what we see as platform-independent
requirements that provide native Swift implementations that aren't burdened
by Apple history.

Hi Dave,

As others have pointed out, we do already have a model for this sort of
thing: the swift server working group.

i feel like the swift server working group is something that is very
difficult to reproduce because in 3 years of Swift it’s the only example of
such a project in existence

That said, there is another analogy which gets closer to what you’re
asking for: the Boost community for C++. Boost was formed because the C++
committee was too bogged down an wasn’t receptive to major library changes
(at one point in time). Boost has effectively parallel leadership from the
C++ committee (though individuals are involved in both organizations of
course). This allows Boost to move faster, ship code, and get experience
with it.

One of the specifically nice things about Boost is that they (at least
originally) focused on building out ideas, getting experience with them,
and then bringing the libraries back to the standard. The libraries
occasionally undergo significant change when they are standardized, but the
usage experience is unmatchable, particularly for very large and complex
APIs.

the problem is boost doesn’t ship with C++ so people view it as just
another dependency, kind of like jQuery but more annoying because you have
to clone and build this giant support library (at least that was my
experience building Blender back in 2012). the C++ STL library or the
python support modules are probably closer to the distribution model I’d
like to see

···

On Thu, Nov 9, 2017 at 1:49 AM, Chris Lattner via swift-evolution < swift-evolution@swift.org> wrote:

> On Nov 7, 2017, at 5:54 PM, Dave DeLong via swift-evolution < > swift-evolution@swift.org> wrote:

In the context of Swift, I think this sort of model could be very
interesting, because there are really several different independent things
going on: for a type like BigNum (for example) there are all the details of
the implementation and design on the one hand, but then there is also the
question of WHICH library it should ship with (Foundation or Swift or
something else). That second decision is much easier to make after the
community has converged on a specific design.

In any case, I think it would be a bad move for the official Swift
distributions to ship code that hasn’t been through the evolution process.
The idea of the Server working group is to delegate detailed design and
iteration to a team of experts, but then have them bring back the API to
evolution when the iteration is done and it is time to “standardize” it. I
think that this is a good model.

-Chris

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[tkremenek (Ted Kremenek)]

Hi Swift-Evolution,

The Standard Library's goal is to be small and targeted. However, many aspects of Apple-provided frameworks need or offer opportunities for improvement or wholesale replacement. These enhancements lie beyond the scope of the Standard Library.

To address this, we'd like to propose the idea of a "Non-Standard Library"; this would be a library that ships with a regular installation of Swift, but is not imported into .swift files by the compiler, unless explicitly requested by the developer.

We are proposing a well-organized effort to parallel the Standard Library without putting additional implementation responsibilities onto the core team. This effort would mitigate what we see as platform-independent requirements that provide native Swift implementations that aren't burdened by Apple history.

Hi Dave,

As others have pointed out, we do already have a model for this sort of thing: the swift server working group.

Hi Chris,

While it is true we have the Swift server work group, it’s not clear that should be the default prototype model to follow here.

That work group was created with a specific purpose to help unify different efforts in the Swift on server community to establish a base set of fundamentals they all could rely upon. But that was after there was already various library implementations/server frameworks in use that could be looked upon to inform technical decisions. The work group is also a fairly heavyweight model; it is appropriate in many contexts but not all of them.

That said, there is another analogy which gets closer to what you’re asking for: the Boost community for C++. Boost was formed because the C++ committee was too bogged down an wasn’t receptive to major library changes (at one point in time). Boost has effectively parallel leadership from the C++ committee (though individuals are involved in both organizations of course). This allows Boost to move faster, ship code, and get experience with it.

One of the specifically nice things about Boost is that they (at least originally) focused on building out ideas, getting experience with them, and then bringing the libraries back to the standard. The libraries occasionally undergo significant change when they are standardized, but the usage experience is unmatchable, particularly for very large and complex APIs.

This “getting experiences with them” model is what appeals to me most about Boost, and is more-or-less what I was suggesting in my earlier reply on this thread as an important ingredient when considering how to extend “core Swift”. The Swift server work group in many ways already follows this model, as it is drawing from other technical work from efforts such as Kitura, Vapor, etc.

In the context of Swift, I think this sort of model could be very interesting, because there are really several different independent things going on: for a type like BigNum (for example) there are all the details of the implementation and design on the one hand, but then there is also the question of WHICH library it should ship with (Foundation or Swift or something else). That second decision is much easier to make after the community has converged on a specific design.

In any case, I think it would be a bad move for the official Swift distributions to ship code that hasn’t been through the evolution process. The idea of the Server working group is to delegate detailed design and iteration to a team of experts, but then have them bring back the API to evolution when the iteration is done and it is time to “standardize” it. I think that this is a good model.

The question in my mind is whether a “work group” is the right default model here. It may be too heavyweight in many cases and perhaps not necessary for smaller efforts or efforts that involve less people.

Also, the server work group has a preconceived plan that the libraries are added to “core Swift”, or in some way become an official part of Swift. I don’t think that needs to be an upfront goal for all Swift libraries out there. The analogy for Boost is interesting; while Boost is a proving ground for many ideas that eventually make it back to the C++ standard, not all of those APIs have that fate and that seems fine too.

Ted

···

On Nov 8, 2017, at 11:49 PM, Chris Lattner via swift-evolution <swift-evolution@swift.org> wrote:

On Nov 7, 2017, at 5:54 PM, Dave DeLong via swift-evolution <swift-evolution@swift.org> wrote:

[taylorswift]

Glad to see people working on this! I already have a functioning
almost-pure Swift PNG library <https://github.com/kelvin13/maxpng&gt;, and a JPEG
library <https://github.com/kelvin13/jpeg&gt; is a month or two away. I’m glad
someone is hacking away at some sort of rasterization engine (like cairo
for C/C++). The way I see a lot of these graphics libraries is they live in
a sort of “tech tree” and a lot of them depend on support higher up the
tree, like in a video game. For now we seem to be filling in a lot of the
top row with C libraries which doesn’t always work too well (like in the
case of zlib).

   Rasterizer File I/O, path manipulation ——————+——— LZ77
compression
[{cairo equiv.}] [{file manager lib}, URL
<https://github.com/kelvin13/url&gt;\] | [{zlib equiv.}]
    > > >
    > +—————————+——————————+————————————+ +——————————————+
    > > > > > > >
    > XML HTML TTF/OTF JPEG PNG GZIP
    > [swiftxml <https://github.com/kelvin13/swiftxml&gt;\] [{???}]
[{???}] [JPEG <https://github.com/kelvin13/jpeg&gt;\] [MaxPNG
<https://github.com/kelvin13/maxpng&gt;\] [{archiving lib}]
    > > > >
    +————————|———————————————+—————+———————————————————+————————————————+
    +——+—————+ | |
       > > >
      SVG Text renderer WOFF
    [{???}] [{freetype equiv.}] [{???}]

···

On Thu, Nov 9, 2017 at 9:25 AM, Benjamin Spratling via swift-evolution < swift-evolution@swift.org> wrote:

Sound great!

Last week I started working on a pure-swift graphics library, one goal
being fast server-side graphics manipulations, and already have .png
decode/encode, and quadratic bezier curve stroking implemented, slowly, and
poorly. I’m working on TrueType fonts right now, and intend to open it to
other developers once the basic architecture is refined. I intend to add
SVG and JPEG support as well. It seems like if there is a “swift-community
blessed geometry library”, this would be a natural extension of that, or at
least depend on it.

Much of this depends on compression support, which I’ve already published
a first cut of at benspratling4 (Ben Spratling IV) · GitHub
SwiftFoundationCompression . The goal there was a more “foundation” idea
of compression than competing libraries, but it suffers from a lack of
streaming support.

I would absolutely love a “BigNum” library. Several encryption packages
out there would benefit from it as well. It seems arithmetic compression
libraries would benefit, too. Other useful Numeric types would include
rational fractions, fixed-point, and short floats (Float16 is what Apple
now thinks the future of images is, right?). I’m happy to contribute; I
end up writing those in a new language ever few years anyway.

-Ben Spratling

On Nov 7, 2017, at 12:54 PM, Dave DeLong via swift-evolution < > swift-evolution@swift.org> wrote:

Hi Swift-Evolution,
...

We propose to create an open-sourced "Non-Standard Library" effort that
coexists with, coordinates with, and is blessed by the open source Swift
development community. The "Non-Standard Library" will offer a
well-curated, high-value collection of routines, types, and documentation
to support Swift development on all platforms.
…

*Suggested Libraries*

There are several areas we think that are ripe for improvement and
implementation as a non-standard library, such as (but not limited to):

   - A BigNum library
   - A full-featured Random library
   - A simplified date/time library
   - A library for manipulating paths (that is not based on URL or String)
   - An expanded Swift-native Geometry library and so forth.

The scope and extent of the sublibraries would be proposed and debated on
a parallel Non-Standard Library Evolution development list.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[davedelong]

As I see it, a “non-standard” library would be present on systems already; you wouldn’t have to bundle it inside your app like you would a third-party library.

But, maybe that’s too high a bar to set, because any library doing that has to start worrying about binary compatibility, in addition to source compatibility.

Dave

···

On Nov 7, 2017, at 3:12 PM, David Sweeris <davesweeris@mac.com> wrote:

On Nov 7, 2017, at 1:58 PM, Ted Kremenek via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Hi Dave,

Thanks for bringing up this topic. This has been kicked around a little, and we’re still exploring different models on how to extend Swift.

The server APIs work group is one operational model for the community to build out a new set of core libraries. That work group was formed out of the observation that there were different implementations of the same thing being created by different Swift on server efforts, and that those different efforts should pool those resources together and standardize on some fundamentals.

That analogy could work here. However, it also has possible major downsides. It can be heavyweight, and also artificially raise the importance of certain people’s library efforts over others by creating a formal work group whose efforts are expected to eventually be incorporated into the core Swift distribution. It would also force a discussion up front of what even makes sense to incorporate into the core Swift distribution, which might be artificially constraining the exploration of the set of libraries to implement.

I need to think about your idea more, but my first reaction is that my preferred route is that the community builds these libraries, ideally using Swift Packages, and through trial and use we evaluate those libraries and then decide if they should be incorporated as part of the core distribution. There’s many factors involved in deciding if they can be incorporated into the core distribution, but all of those could be informed by actually building libraries and see them getting used.

We could also figure out a way to possibly highlight these efforts to the Swift community, maybe on swift-evolution or other means — but all with the expectation that these libraries are not *necessarily* going to be standardized as part of the core swift distribution. I don’t think that’s all that bad either; not every library will make sense to incorporate into the core Swift distribution (e.g., they are highly domain specific) but still supporting their development would be beneficial to the community.

Note that any change going into the Swift core distribution inevitably involves swift-evolution and the core team. Changes going into the core Swift distribution must meet a very high bar as far as implementation and design, the confidence we have into committing to specific APIs (especially since we’ll have ABI stability in Swift 5), and other factors. Thus this will not really alleviate much burden on the Core team or on the community — one of the core goals of your proposal. Further, incorporating all those concerns up front when building libraries might be counterproductive.

FWIW, Ben Cohen and I have been talking about possibly using Swift packages as a way to seed out experimental ideas for extensions to the Standard Library. This would allow ideas to be trialed by real usage (a complaint I’ve seen about some changes we’ve made to Swift in the past). Users could build things on top of those libraries, knowing they are available as packages, and if an API “graduates” to being part of the Standard Library the user can then depend upon it being available there. If it never graduates, however, the package remains around.

One thing that resonates me in what you propose is about having a “well-organized effort” whose aim is to make these libraries feel cohesive and implemented well. However, given that many of these naturally fall somewhere in the spectrum of responsibilities within the Standard Library and Foundation, I think it is self-misleading to think that the Core Team or others would not be involved in these efforts if the intention is to possibly incorporate these one day into the core Swift distribution. There also may be other ways to achieve that level of cohesion in API design, such as through community discussion (possibly via the Swift.org <http://swift.org/&gt; mailing lists/forums). This discussion would not necessarily be the same as the path to “ratification” of a library into core Swift, but a step that could benefit many library authors (including considering ideas one day incorporated into the core swift). With such a mechanism many library authors could benefit even if they do not intend to have the library as part of the core distribution.

My apologies that these are all hand-wavy ideas, but I’m trying to paint a possible alternative way to achieve your goal of more library evolution for Swift without having such a formal work group that may be too heavy weight or too narrowly focused.

FWIW, I don't think it much matters if they get rolled into the "core" Swift distribution or library... I think all that's really necessary (well, to the extent that any of this is) is just to have an easy way to search for & install packages. Even just having the commonly used ones bundled into a "not quite standard library" that's a separate download would be icing on the cake, IMHO, because it makes it easy to find them and know that with them being widely used, they're probably at least reasonably fit for purpose.

- Dave Sweeris

[Wallacy]

@Félix Fischer

Yes, I wanted to say central index. I chose the wrong term.

Actually I could have summed it up like this:
- Central Index
- SwiftPM to download/search using this index like npm/nuget
- GStreamer organization Style.

I do not agree with Ted that only a few projects should be ranked,
everyone, as it is in npm should be available. Only be graded according to
recommendations.

For exemple this is what GStreamer do:

*gst-plugins-base*
a small and fixed set of plug-ins, covering a wide range of possible types
of elements; these are continuously kept up-to-date with any core changes
during the development series.

   - We believe distributors can safely ship these plug-ins
   - People writing elements should base their code on these elements
   - These elements come with examples, documentation, and regression tests

*gst-plugins-good*
a set of plug-ins that we consider to have good quality code, correct
functionality, our preferred license (LGPL for the plug-in code, LGPL or
LGPL-compatible for the supporting library).

   - We believe distributors can safely ship these plug-ins
   - People writing elements should base their code on these elements

*gst-plugins-ugly*
a set of plug-ins that have good quality and correct functionality, but
distributing them might pose problems. The license on either the plug-ins
or the supporting libraries might not be how we'd like. The code might be
widely known to present patent problems.

   - Distributors should check if they want/can ship these plug-ins
   - People writing elements should base their code on these elements

*gst-plugins-bad*
a set of plug-ins that aren't up to par compared to the rest. They might be
close to being good quality, but they're missing something - be it a good
code review, some documentation, a set of tests, a real live maintainer, or
some actual wide use. If the blanks are filled in they might be upgraded to
become part of either gst-plugins-good or gst-plugins-ugly, depending on
the other factors.

   - If the plug-ins break, you can't complain - instead, you can fix the
   problem and send us a patch, or bribe someone into fixing them for you
   - New contributors can start here for things to work on

For example, let's say that the proposal on type Result <T> is not approved.

The community cam build a Result<T> "project" ate "swift-extension-base"
and everyone at "base" level must agree to use then (or improve then) to
avoid one of the concerns at Result<T> proposal (several projects build the
own)...

So in Swift side maybe the indexer (and the community) can classify
projects in this way:

*swift-extension-base *// (Swift Core Team maybe can manager this one)

*a small and fixed set of features, covering a small range of possible
types of elements*; these are continuously kept up-to-date with any core
changes during the development series.

   - We believe distributors can safely ship these features
   - People writing elements should base their code on these elements
   - These elements come with examples, documentation, and regression tests

*swift-extension-good *// (Community can manage this, and Swift Core Team
can do some advices)
a set of *features *that we consider to have good quality code, correct
functionality, our preferred license* (Swift Licence)*, and *covering a
wide range features.*

   - We believe distributors can safely ship these plug-ins
   - People writing elements should base their code on these elements

*swift-extension-ugly *// (Full Community)
a set of *features *that have good quality and correct functionality, but
distributing them might pose problems. The license on either the or the
supporting libraries might not be how we'd like. The code might be widely
known to present patent problems.

   - Distributors should check if they want/can ship these plug-ins
   - People writing elements should base their code on these elements

*swift-extension-bad *// (Full Community)
a set of *features *that aren't up to par compared to the rest. They might
be close to being good quality, but they're missing something - be it a
good code review, some documentation, a set of tests, a real live
maintainer, or some actual wide use.

   - If the *features *break, you can't complain - instead, you can fix the
   problem and send us a patch, or bribe someone into fixing them for you
   - New contributors can start here for things to work on

I just change plugins to feature to make more sense here... And at "base"
level we can just drop very small projects, and make few guarantees that
will always work. And at good level we can drop bigger projects like
Alamofire, Perfect etc, because of the nature of those projects, we know
that will get a good support but we cant guarantee that documentation and
tests and language updates will always be pair with the rest.

And at "bad" level, will be like NPM is today... just a easy way to obtain
some 3rd party code but no guarantees at all!

We heve IBM Swift Package Catalog (https://packagecatalog.com) but i think
we can do better.

Also, if we (and Apple) do nothing, the community someway will do! It's
like Brew/Macports at some point people will build the tools to make the
ecosystem easier. Is better make something oficial (like npm/nuget) of
course.

A "Non-Standard Libraries" its a non start for me, but good indexer, a good
integration with SwiftPM, a oficial help/support of the Swift Core Team,
and a set of rules/patterns to rule this (and other) projects is a good
feature to have.

···

Em ter, 7 de nov de 2017 às 21:16, Alejandro Martinez via swift-evolution < swift-evolution@swift.org> escreveu:

I’m with Ted on this one.
I would also love to see the Swift ecosystem grow but I think that it has
to happen with SPM. With improvements on SPM (as discussed in other
threads) and having a proper index (imho Cocoapods webpage is the best one
out there, with stats about docs, unit testing, downloads...) that makes
finding libraries easier.
But also a discussion forum where great libs can be shared and highlighted
to the rest of the community and big community efforts coordinated. If you
look at Rust is a great example, their forums attract the whole community
(sadly not something the mailing list does) and big and important projects
have come up from the community that have made a huge impact. With that in
place and some libs out there maturing I’m sure that in the future the
conversation to include battle tested code into an oficial distribution
would be easier.

Sent from my iPad

On 7 Nov 2017, at 21:58, Ted Kremenek via swift-evolution < > swift-evolution@swift.org> wrote:

Hi Dave,

Thanks for bringing up this topic. This has been kicked around a little,
and we’re still exploring different models on how to extend Swift.

The server APIs work group is one operational model for the community to
build out a new set of core libraries. That work group was formed out of
the observation that there were different implementations of the same thing
being created by different Swift on server efforts, and that those
different efforts should pool those resources together and standardize on
some fundamentals.

That analogy could work here. However, it also has possible major
downsides. It can be heavyweight, and also artificially raise the
importance of certain people’s library efforts over others by creating a
formal work group whose efforts are expected to eventually be incorporated
into the core Swift distribution. It would also force a discussion up
front of what even makes sense to incorporate into the core Swift
distribution, which might be artificially constraining the exploration of
the set of libraries to implement.

I need to think about your idea more, but my first reaction is that my
preferred route is that the community builds these libraries, ideally using
Swift Packages, and through trial and use we evaluate those libraries and
then decide if they should be incorporated as part of the core
distribution. There’s many factors involved in deciding if they can be
incorporated into the core distribution, but all of those could be informed
by actually building libraries and see them getting used.

We could also figure out a way to possibly highlight these efforts to the
Swift community, maybe on swift-evolution or other means — but all with the
expectation that these libraries are not *necessarily* going to be
standardized as part of the core swift distribution. I don’t think that’s
all that bad either; not every library will make sense to incorporate into
the core Swift distribution (e.g., they are highly domain specific) but
still supporting their development would be beneficial to the community.

Note that any change going into the Swift core distribution inevitably
involves swift-evolution and the core team. Changes going into the core
Swift distribution must meet a very high bar as far as implementation and
design, the confidence we have into committing to specific APIs (especially
since we’ll have ABI stability in Swift 5), and other factors. Thus this
will not really alleviate much burden on the Core team or on the community
— one of the core goals of your proposal. Further, incorporating all those
concerns up front when building libraries might be counterproductive.

FWIW, Ben Cohen and I have been talking about possibly using Swift
packages as a way to seed out experimental ideas for extensions to the
Standard Library. This would allow ideas to be trialed by real usage (a
complaint I’ve seen about some changes we’ve made to Swift in the past).
Users could build things on top of those libraries, knowing they are
available as packages, and if an API “graduates” to being part of the
Standard Library the user can then depend upon it being available there.
If it never graduates, however, the package remains around.

One thing that resonates me in what you propose is about having a
“well-organized effort” whose aim is to make these libraries feel cohesive
and implemented well. However, given that many of these naturally fall
somewhere in the spectrum of responsibilities within the Standard Library
and Foundation, I think it is self-misleading to think that the Core Team
or others would not be involved in these efforts if the intention is to
possibly incorporate these one day into the core Swift distribution. There
also may be other ways to achieve that level of cohesion in API design,
such as through community discussion (possibly via the Swift.org mailing
lists/forums). This discussion would not necessarily be the same as the
path to “ratification” of a library into core Swift, but a step that could
benefit many library authors (including considering ideas one day
incorporated into the core swift). With such a mechanism many library
authors could benefit even if they do not intend to have the library as
part of the core distribution.

My apologies that these are all hand-wavy ideas, but I’m trying to paint a
possible alternative way to achieve your goal of more library evolution for
Swift without having such a formal work group that may be too heavy weight
or too narrowly focused.

Ted

On Nov 7, 2017, at 9:54 AM, Dave DeLong via swift-evolution < > swift-evolution@swift.org> wrote:

Hi Swift-Evolution,

The Standard Library's goal is to be small and targeted. However, many
aspects of Apple-provided frameworks need or offer opportunities for
improvement or wholesale replacement. These enhancements lie beyond the
scope of the Standard Library.

To address this, we'd like to propose the idea of a "Non-Standard
Library"; this would be a library that ships with a regular installation of
Swift, but is not imported into .swift files by the compiler, unless
explicitly requested by the developer.

We are proposing a well-organized effort to parallel the Standard Library
without putting additional implementation responsibilities onto the core
team. This effort would mitigate what we see as platform-independent
requirements that provide native Swift implementations that aren't
burdened by Apple history.

*Mission Statement*

We propose to create an open-sourced "Non-Standard Library" effort that
coexists with, coordinates with, and is blessed by the open source Swift
development community. The "Non-Standard Library" will offer a
well-curated, high-value collection of routines, types, and documentation
to support Swift development on all platforms.

*Goals*

The main goals of this effort would be:

   - Alleviate pressure on the Core Team while providing the developer
   community with functionality that normally falls under Apple's internal
   development umbrella.
   - Deliver authoritative, reliable, and regularly updated libraries
   avoiding issues faced by third-party dependencies.
   - Provide oversight, organization, and full community involvement to
   ensure its components are worthy, maintained, and passing a bar of need and
   excellence.

*Suggested Libraries*

There are several areas we think that are ripe for improvement and
implementation as a non-standard library, such as (but not limited to):

   - A BigNum library
   - A full-featured Random library
   - A simplified date/time library
   - A library for manipulating paths (that is not based on URL or String)
   - An expanded Swift-native Geometry library and so forth.

The scope and extent of the sublibraries would be proposed and debated on
a parallel Non-Standard Library Evolution development list.

*Coordination*

This effort would be fully coordinated with the Swift development team,
respecting the team's existing commitments and responsibilities. We would
request an Apple body to act as an official coordinator, enabling both
oversight of this effort and to expose Apple-sourced suggestions to
the Non-Standard community for action.

*Next Steps*

To proceed, we need a general community consensus that this effort is
worth the significant overhead it would involve.

We must then:

   - Select a project lead, who appoints technical leaders from the
   community.
   - Recruit a core team, a small group responsible for strategic
   direction, pulled from experienced members well versed in contributing to
   Swift-dev.
   - Establish a Non-Standard Swift Evolution process, based on the one
   that is currently followed by Swift Evolution. Following SE practices will
   guarantee a consistent and excellent language experience for those
   developers including the Non-Standard library into their projects.
   - Build a Non-Standard Swift Evolution repository home.
   - Adopt a code license, based on Swift's current licensing.
   - Define the community working group rules and code of conduct.
   - Establish a mailing list and/or forum.
   - Begin the selection of target sublibraries and populating them by
   recruiting committers and contributors.

*Alternative Names*

Suggested names for this effort include the following.

   - Extended-Standard Library
   - Community-Standard Library
   - Non-Standard Library

We are not married to any of these names and welcome alternative
suggestions.

*Measures of Success*

A successful Non-Standard Library will provide a stable and incrementally
grown ecology of Swift frameworks that developers can depend upon with a
minimum of turbulence and regret cycles. For that reason, the most
successful content will be core functionality, with significant field
testing prior to adoption. We recommend that NSSE follow a model of
provisionary adoption and refinement before deployment to ensure any
adopting code base will not be affected by later module changes.

Thanks,

Dave DeLong and Erica Sadun
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[nick.keets]

This is a great idea. Enabling people to actually sell code could be game
changing.

···

On Wed, Nov 8, 2017 at 12:44 AM, Jonathan Hull via swift-evolution < swift-evolution@swift.org> wrote:

On Nov 7, 2017, at 1:58 PM, Ted Kremenek via swift-evolution < > swift-evolution@swift.org> wrote:

We could also figure out a way to possibly highlight these efforts to the
Swift community, maybe on swift-evolution or other means — but all with the
expectation that these libraries are not *necessarily* going to be
standardized as part of the core swift distribution. I don’t think that’s
all that bad either; not every library will make sense to incorporate into
the core Swift distribution (e.g., they are highly domain specific) but
still supporting their development would be beneficial to the community.

What I *really* want to see is a Code Store in Xcode (kind of like an app
store for Swift packages). Basically, you would hit “Add Package” in the
sidebar/menu and a sheet pops up where you can search for and view various
packages. You then hit an “Add” button in the sheet, and it would add the
package to your project. To remove a package you would select it and hit
delete.

Selecting a package in the sidebar would let you set some settings (e.g.
select a specific version), and view documentation. Xcode would manage the
files for you behind the scenes.

I know that is out of scope for Evolution, but I just wanted to mention
it, since it might help with the visibility issue.

Thanks,
Jon

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[tkremenek (Ted Kremenek)]

Right, and that would be the case with anything that is actually vended as part of the core distribution. I’d like to see more great libraries being built that have high value to the community without having to have this artificially high bar set from the onset.

···

On Nov 7, 2017, at 2:14 PM, Dave DeLong <swift@davedelong.com> wrote:

On Nov 7, 2017, at 3:12 PM, David Sweeris <davesweeris@mac.com> wrote:

On Nov 7, 2017, at 1:58 PM, Ted Kremenek via swift-evolution <swift-evolution@swift.org> wrote:

Hi Dave,

Thanks for bringing up this topic. This has been kicked around a little, and we’re still exploring different models on how to extend Swift.

The server APIs work group is one operational model for the community to build out a new set of core libraries. That work group was formed out of the observation that there were different implementations of the same thing being created by different Swift on server efforts, and that those different efforts should pool those resources together and standardize on some fundamentals.

That analogy could work here. However, it also has possible major downsides. It can be heavyweight, and also artificially raise the importance of certain people’s library efforts over others by creating a formal work group whose efforts are expected to eventually be incorporated into the core Swift distribution. It would also force a discussion up front of what even makes sense to incorporate into the core Swift distribution, which might be artificially constraining the exploration of the set of libraries to implement.

I need to think about your idea more, but my first reaction is that my preferred route is that the community builds these libraries, ideally using Swift Packages, and through trial and use we evaluate those libraries and then decide if they should be incorporated as part of the core distribution. There’s many factors involved in deciding if they can be incorporated into the core distribution, but all of those could be informed by actually building libraries and see them getting used.

We could also figure out a way to possibly highlight these efforts to the Swift community, maybe on swift-evolution or other means — but all with the expectation that these libraries are not *necessarily* going to be standardized as part of the core swift distribution. I don’t think that’s all that bad either; not every library will make sense to incorporate into the core Swift distribution (e.g., they are highly domain specific) but still supporting their development would be beneficial to the community.

Note that any change going into the Swift core distribution inevitably involves swift-evolution and the core team. Changes going into the core Swift distribution must meet a very high bar as far as implementation and design, the confidence we have into committing to specific APIs (especially since we’ll have ABI stability in Swift 5), and other factors. Thus this will not really alleviate much burden on the Core team or on the community — one of the core goals of your proposal. Further, incorporating all those concerns up front when building libraries might be counterproductive.

FWIW, Ben Cohen and I have been talking about possibly using Swift packages as a way to seed out experimental ideas for extensions to the Standard Library. This would allow ideas to be trialed by real usage (a complaint I’ve seen about some changes we’ve made to Swift in the past). Users could build things on top of those libraries, knowing they are available as packages, and if an API “graduates” to being part of the Standard Library the user can then depend upon it being available there. If it never graduates, however, the package remains around.

One thing that resonates me in what you propose is about having a “well-organized effort” whose aim is to make these libraries feel cohesive and implemented well. However, given that many of these naturally fall somewhere in the spectrum of responsibilities within the Standard Library and Foundation, I think it is self-misleading to think that the Core Team or others would not be involved in these efforts if the intention is to possibly incorporate these one day into the core Swift distribution. There also may be other ways to achieve that level of cohesion in API design, such as through community discussion (possibly via the Swift.org mailing lists/forums). This discussion would not necessarily be the same as the path to “ratification” of a library into core Swift, but a step that could benefit many library authors (including considering ideas one day incorporated into the core swift). With such a mechanism many library authors could benefit even if they do not intend to have the library as part of the core distribution.

My apologies that these are all hand-wavy ideas, but I’m trying to paint a possible alternative way to achieve your goal of more library evolution for Swift without having such a formal work group that may be too heavy weight or too narrowly focused.

FWIW, I don't think it much matters if they get rolled into the "core" Swift distribution or library... I think all that's really necessary (well, to the extent that any of this is) is just to have an easy way to search for & install packages. Even just having the commonly used ones bundled into a "not quite standard library" that's a separate download would be icing on the cake, IMHO, because it makes it easy to find them and know that with them being widely used, they're probably at least reasonably fit for purpose.

- Dave Sweeris

As I see it, a “non-standard” library would be present on systems already; you wouldn’t have to bundle it inside your app like you would a third-party library.

But, maybe that’s too high a bar to set, because any library doing that has to start worrying about binary compatibility, in addition to source compatibility.

[tkremenek (Ted Kremenek)]

I’m a bit confused. I’m not sure what comments of mine I’m referring to. The main thing I said is that I’d prefer the community grow libraries that are trialed and used first (and then possibly incorporated into “core Swift”) rather than focus on standardizing a set of new libraries for distribution with Swift from the onset.

···

On Nov 8, 2017, at 4:30 AM, Wallacy via swift-evolution <swift-evolution@swift.org> wrote:

I do not agree with Ted that only a few projects should be ranked, everyone, as it is in npm should be available. Only be graded according to recommendations.

[tkremenek (Ted Kremenek)]

Clearly I’m double confused. That meant to read “I’m not sure what comments of mine *you* are referring to”.

I fully support having a broad spectrum of libraries that the community builds and uses. Any library that we decide to make part of “core Swift” — IMHO at a mature point in a library’s evolution — would need to have high value to the majority of the community and would need to feel solid enough that we can lock it in for both source and binary compatibility, high quality of implementation with sustained maintenance, etc.

···

On Nov 8, 2017, at 11:40 AM, Ted Kremenek via swift-evolution <swift-evolution@swift.org> wrote:

On Nov 8, 2017, at 4:30 AM, Wallacy via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

I do not agree with Ted that only a few projects should be ranked, everyone, as it is in npm should be available. Only be graded according to recommendations.

I’m a bit confused. I’m not sure what comments of mine I’m referring to.

[Karl]

Yeah this is exactly the problem that the package manager is there to solve, right? It’s supposed to make it ridiculously easy to integrate libraries and manage your dependencies.

The problem is that most people writing Swift code every day are doing it to make graphical applications on iOS/macOS. SwiftPM doesn’t support those, so if I want to test a library, it’s just a one-off thing that I play with in a Playground.

I think that the best thing we could do to encourage people to write, use and contribute to public libraries would be to improve the package manager. SwiftPM is still basically a toy (or an interesting curiosity), until it can actually be used in the projects most Swift devs get paid to work on every day. Talking about it supporting a community is way premature; it’s not even close to ready to taking on that responsibility, IMO.

- Karl

···

On Nov 7, 2017, at 1:58 PM, Ted Kremenek via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

FWIW, Ben Cohen and I have been talking about possibly using Swift packages as a way to seed out experimental ideas for extensions to the Standard Library. This would allow ideas to be trialed by real usage (a complaint I’ve seen about some changes we’ve made to Swift in the past). Users could build things on top of those libraries, knowing they are available as packages, and if an API “graduates” to being part of the Standard Library the user can then depend upon it being available there. If it never graduates, however, the package remains around.

[taylorswift]

i mean I don’t think these approaches are incompatible. The “swift core”
could just make the process of independent libraries getting started
easier. Like right now there’s really no place to say “hey I just started a
library project for X, and anyone who wants to be involved should
contribute at Y github repo where it lives right now”. I’ve tried sending
that on this list before and it didn’t really work because mailing lists
aren’t really a good medium for that and no one wants the swift-evolution
list getting clogged with project-specific messages most people don’t care
about.

···

On Wed, Nov 8, 2017 at 1:58 PM, Ted Kremenek via swift-evolution < swift-evolution@swift.org> wrote:

On Nov 8, 2017, at 11:40 AM, Ted Kremenek via swift-evolution < > swift-evolution@swift.org> wrote:

On Nov 8, 2017, at 4:30 AM, Wallacy via swift-evolution < > swift-evolution@swift.org> wrote:

I do not agree with Ted that only a few projects should be ranked,
everyone, as it is in npm should be available. Only be graded according to
recommendations.

I’m a bit confused. I’m not sure what comments of mine I’m referring to.

Clearly I’m double confused. That meant to read “I’m not sure what
comments of mine *you* are referring to”.

I fully support having a broad spectrum of libraries that the community
builds and uses. Any library that we decide to make part of “core Swift” —
IMHO at a mature point in a library’s evolution — would need to have high
value to the majority of the community and would need to feel solid enough
that we can lock it in for both source and binary compatibility, high
quality of implementation with sustained maintenance, etc.