Trajectory for evaluating adding Automatic Differentiation to Swift

[tkremenek (Ted Kremenek)]

One of the efforts of the Swift for TensorFlow project has been to explore adding features like Automatic Differentiation to the Swift language. This is a powerful capability that can significantly enrich Swift’s potential as a programming language for scientific computing, numerics, and machine learning.

Speaking on behalf of the Swift Core Team, we are interested in evaluating incorporating this capability directly into the Swift language.

As usual, changes to Swift must go through the Swift Evolution process. Further, any proposed change requires an implementation that can be evaluated and understood with its full impact on both (a) the feel of the Swift language and (b) the language’s implementation.

There are various ways to incubate an implementation of proposed changes to Swift. This includes using branches or having the implementation directly on master behind various kinds of feature gates. Both approaches have been used in the past. For Automation Differentiation, the Core Team is supportive of using the second approach given the deep tie-ins with the rest of the compiler, the likely rapid iteration of the work, and that overall development will benefit from testing and core development being done in master rather than a long-lived parallel branch.

To summarize:

1. An implementation of Automatic differentiation will be added to the compiler, guarded under a flag (or flags) to indicate it is an experimental feature. This will be done directly on master.

2. Once implementations are ready, each component of the feature will go through the Swift Evolution process. Until that time, the experimental feature will not be included in official Swift releases.

The implementation of Automatic Differentiation is likely to touch on the compiler, runtime, and Standard Library types. As the experimental implementation is staged in, we’ll figure out how to best factor the experimental nature of this feature into the impacted components.

[dan-zheng]

Thank you Ted!

@rxwei earlier wrote an upstreaming plan that lays out the process in detail. To summarize: differentiable programming features will be flag-gated and tested on master, and development will continue on master after upstreaming is done. We will request code review from code owners on individual upstreaming PRs.

Please reply if you have questions or feedback!

[fbartho (Frederic Barthelemy)]

Is there any "sketch" idea of how this might be integrated at the Syntax level? I'm super curious to understand what this will involve, and why it's getting fast-tracked in.

I'm new to Automatic Differentiation, but it seems like a purpose specific symbolic math library? Why is it appropriate to add this syntactically to the language? Wouldn't it be better done as a free-standing library? Since it has major impact on the compiler, parser, other things that really seems like something that we shouldn't rush in dropping in?

Thanks!

[rex-remind (Rex)]

Hello, these sections in the mega-proposal might help. Motivation and some history on how Automatic Differentiation differentiates.. itself from Symbolic Differentiation.

[dan-zheng]

Hi Frederic,

Thanks for raising your concern! Our differentiable programming manifesto (written in a Swift Evolution format) may help give you some context on our vision. I'd recommend the following sections in particular:

• "Approaches to automatic differentiation": we mention existing approaches to automatic differentiation, including library-based approaches, source code transformation tools, and language integration.
• "Why bake differentiation into Swift?": we make the case that first-class language support for differentiation will enable convenient, extensible, and performant differentiable programming in Swift.

The forum thread below has more discussions, if you're curious:

Differentiable Programming Mega-Proposal

Hi Swift community, We have completed a comprehensive proposal for the differentiable programming feature we’ve been incubating over the last 1.5 years. We’ve gone over many iterations on the feature design, and have partially completed the implementation. Now we are ready to start a discussion on Swift Evolution, specifically on upstreaming and standardizing the feature. Since this proposal is overly long (~60 pages), we hope to start by merging it into the docs/ directory in apple/swift via a…

[Troy_Harvey]

I think this was well covered in previous discussion. But to summarize:

1. Generalized auto-diff, at least in a statically analyzed Swift view of the world, could only be done in the compiler. A library isn’t a viable solution for the goals of the project.

2. This is just math. It’s general. No different then any other base math function, like cos, tan, or abs. It is a quirk of computing history that it took us 40 years to get it into a language proper next to its math function peers (well that and its a lot harder)

3. I don’t think this is fast tracked. This is 1.5 years in the making. One of the phenomenal things about Swift, is it has 2 large corporate backers. Each that brings a different view and experience. It is an incredible opportunity to unify system and app development, with scripting, ML and data science under one roof.

[dan-zheng]

Update: ~all code related to differentiable programming in Swift has been upstreamed to apple/swift master. Differentiable programming features are available from master toolchains, gated by import _Differentiation:

import _Differentiation
import Darwin

func foo(_ x: Float) -> Float {
  cos(x) + sin(x)
}
print(gradient(at: 10, in: foo)) // -0.29505038

Projects like Swift for TensorFlow, apple/swift-numerics, and borglab/SwiftFusion are using and experimenting with these features.

We're continuing differentiable programming development on apple/swift master branch, and we'll upstream remaining AutoDiff tests over time.

---

It took us a few months to upstream all our code from apple/swift tensorflow branch to master branch.

Thank you to everyone who helped review our code! Folks like @codafi and @Michael_Gottesman gave much-appreciated suggestions on various code improvements - we'll stay in touch and work on addressing them.

Currently, the Swift Differentiable Programming Manifesto remains our main documentation. We plan to start writing usage documentation soon, as well as posting monthly progress updates with the #autodiff tag. As @rxwei says, maybe differentiable programming will be ready for Swift Evolution next year.

Let us know if you have any questions. Cheers!

[ExFalsoQuodlibet (Elviro Rocca)]

This is amazing, thanks a lot for all your work on this.

[dan-zheng]

Thank you!

I'd like to recognize "us", the Differentiable Swift authors: @rxwei, @marcrasi, @bartchr808, me. You may see us talking about differentiable programming on the forums or GitHub.

Thanks also to Swift for TensorFlow team members for providing support.