One of the efforts of the Swift for TensorFlow project has been to explore adding features like Automatic Differentiation to the Swift language. This is a powerful capability that can significantly enrich Swift’s potential as a programming language for scientific computing, numerics, and machine learning.
Speaking on behalf of the Swift Core Team, we are interested in evaluating incorporating this capability directly into the Swift language.
As usual, changes to Swift must go through the Swift Evolution process. Further, any proposed change requires an implementation that can be evaluated and understood with its full impact on both (a) the feel of the Swift language and (b) the language’s implementation.
There are various ways to incubate an implementation of proposed changes to Swift. This includes using branches or having the implementation directly on master behind various kinds of feature gates. Both approaches have been used in the past. For Automation Differentiation, the Core Team is supportive of using the second approach given the deep tie-ins with the rest of the compiler, the likely rapid iteration of the work, and that overall development will benefit from testing and core development being done in master rather than a long-lived parallel branch.
An implementation of Automatic differentiation will be added to the compiler, guarded under a flag (or flags) to indicate it is an experimental feature. This will be done directly on master.
Once implementations are ready, each component of the feature will go through the Swift Evolution process. Until that time, the experimental feature will not be included in official Swift releases.
The implementation of Automatic Differentiation is likely to touch on the compiler, runtime, and Standard Library types. As the experimental implementation is staged in, we’ll figure out how to best factor the experimental nature of this feature into the impacted components.
@rxwei earlier wrote an upstreaming plan that lays out the process in detail. To summarize: differentiable programming features will be flag-gated and tested on master, and development will continue on master after upstreaming is done. We will request code review from code owners on individual upstreaming PRs.
Is there any "sketch" idea of how this might be integrated at the Syntax level? I'm super curious to understand what this will involve, and why it's getting fast-tracked in.
I'm new to Automatic Differentiation, but it seems like a purpose specific symbolic math library? Why is it appropriate to add this syntactically to the language? Wouldn't it be better done as a free-standing library? Since it has major impact on the compiler, parser, other things that really seems like something that we shouldn't rush in dropping in?
Thanks for raising your concern! Our differentiable programming manifesto (written in a Swift Evolution format) may help give you some context on our vision. I'd recommend the following sections in particular:
I think this was well covered in previous discussion. But to summarize:
Generalized auto-diff, at least in a statically analyzed Swift view of the world, could only be done in the compiler. A library isn’t a viable solution for the goals of the project.
This is just math. It’s general. No different then any other base math function, like cos, tan, or abs. It is a quirk of computing history that it took us 40 years to get it into a language proper next to its math function peers (well that and its a lot harder)
I don’t think this is fast tracked. This is 1.5 years in the making. One of the phenomenal things about Swift, is it has 2 large corporate backers. Each that brings a different view and experience. It is an incredible opportunity to unify system and app development, with scripting, ML and data science under one roof.
Update: ~all code related to differentiable programming in Swift has been upstreamed to apple/swift master. Differentiable programming features are available from master toolchains, gated by import _Differentiation:
Thank you to everyone who helped review our code! Folks like @codafi and @Michael_Gottesman gave much-appreciated suggestions on various code improvements - we'll stay in touch and work on addressing them.