I'm really excited to see this work start to make its way into the official SE process. I can't wait to see the applications for this the Swift community develops in the coming years.
Thank you for all the hard work you're doing to make Swift a great language in new domains! It is important for Swift to continue to grow beyond its initial domain of building apps for Apple platforms. I can't imagine a better way to do that than giving Swift a unique and innovative capability that is extremely relevant to the industry today. Even better, this feature targets a domain focused on high performance numerical computing. Making Swift a great language for numerical computing will benefit the whole community by letting us do more without leaving the language (especially since having the language often means dropping down to C and losing memory safety).
Language vs library
Since there are no precedents for a language with first class automatic differentiation it is no surprise that there is some pushback on deep compiler integration. This is especially the case given Swift's relative weakness in the area of metaprogramming and absence of support for library-defined code transformations.
While it may be possible to move some parts of this feature into a library it looks to me like requesting that the entire feature live in a library isn't reasonable given the stated goal. The design is deeply integrated with the language and the type system. This integration is essential to providing truly first class AD. As an example, it is not at all clear to me how a library could do replace @differentiable function types without imposing a significant burden on users of AD, especially given the carefully thought out subtyping relationships of such functions.
I would be interested in seeing the authors explore this topic a bit deeper. In an ideal world, what metaprogramming features would you like to see in Swift to support AD and how would that influence the design of this proposal? What parts do you think could ideally live in a library and which features really must be implemented in language itself?
I agree with the concern of others have expressed that there is a risk of continuing to kick metaprogramming down the road if when we encounter large concrete use cases for metaprogramming features, instead of designing and implementing them we put special case features in the language that might someday be replaced with a library solution building on top of future metaprogramming features. We should be cognizant of this risk and honest about how far we go down this road before we start prioritizing the metaprogramming tools Swift needs in order to enable library developers.
AD beyond ML
Other pushback appears to be due to the roots of this feature in the ML domain and a sense that the feature won't really general purpose in practice. It's natural that many of the concrete examples in the proposal are derived from the domain the team building it is focused on. This gives the proposal an emphasis on ML that I don't believe is representative of how the Swift community will use the feature if it is accepted. Of course it will be heavily used in the ML domain, but it will also be used in interesting ways in many other domains.
With no precedent in other languages we aren't able to draw on concrete examples that have been developed elsewhere in other domains. We have to use our imaginations. I think the proposal does a good job of describing domains beyond ML where AD will be extremely useful, such as graphics and scientific computing. But pointing in a direction is much different than providing concrete examples. I think the authors could make a stronger case for the general utility of the feature by fleshing out even just a few simple examples in other domains.
I also encourage people pushing back to think not only about their own code, but also about the libraries and frameworks that might be interesting to them which might take advantage of AD. We all benefit (at least indirectly) when Swift becomes a better language for writing libraries.
Additional language integration
The proposal does not discuss any interaction of AD with Swift's mutability model. There is a close relationship between a function with the signature (Float) -> Float and one with the signature (inout Float) -> Void. Is there a reason AD supports the former and not the latter? Is deeper integration with the mutability model something that could be explored in the future?
Similarly, the proposal mentions the possibility of providing a condition conformance for Result to Differentiable. There is a close relationship between (Float) -> Result<Float, E> (which could be differentiable when E: Differentiable and (Float) throws -> Float. If typed errors are added to Swift in the future would it be reasonable to consider supporting differentiation of throwing functions when the error type conforms to Differentiable? The utility of this is not immediately obvious so feel free to consider it a hypothetical question.
Minor questions
If a non- Differentiable or a let stored property is not marked with @noDerivative , then it is treated as if it has @noDerivative and the compiler emits a warning (with a fix-it in IDEs) asking the user to make the attribute explicit.
Why is this a warning rather than an error? It seems hard to justify impact behavior with a warning. We don't do that anywhere else in Swift (afaik).
Properties in the synthesized TangentVector have the same effective access level as their corresponding original properties.
Can you clarify what you mean by the same effective access level when the original property was declared private? The original property would be private to the Differentiable type within the declaring file. There is no way to spell that scope of visibility on a property of a different type. Would the property be private to the synthesized TangentVector, fileprivate (so it is also visible in the scope of the Differentiable type in the declaring file), or would it be a Voldemort scope of some kind?
)