Differentiable programming for gradient-based machine learning

That is definitely an important use case, and it could help Swift autodiff/S4TF stand out from other languages/frameworks. @rxwei we now have complex physics simulations and biomedicine as additional major use cases for autodiff.

1 Like

@binderh have you checked out my thread on enabling differentiation in Swift release toolchains? Swift for TensorFlow Resurrection: Differentiation running on iOS

(post deleted by author)

3 Likes

I feel great respect to your eternal fight efforts in this area.

Don't take this as a discouragement, just one humble opinion of a member of this list.

Have no idea how others feel about this. My view on this and other features like this:

  • Swift is already quite large and complicated language.
  • We can't add everything into Swift to satisfy everyone.
  • There are costs to every new language feature (compiler complexity, app complexity, user education, etc)
  • Swift should stay general purpose language without extending into niche domains and areas that would be only used in, say, 0.1% of all apps.
  • Can a feature be done in a library form (even if it's not totally perfect that way compared to being built into the language itself)? If yes it should stay in a form of separate library.

All IMHO above, and I'm happy to be proven wrong.

To be clear, the current stance of the Swift project towards this feature is still what Ted laid out in his post from 2019. There's technical progress that needs to be made, and then we can evaluate adding this to the language. Philip's work is welcome, but this is a large feature that will be ready when it's ready, and it is not on the roadmap for any particular release.

There probably isn't a whole lot of point to discussing this feature in the abstract right now. Implementation progress and discussion should really go into a dedicated thread in Development, and nothing is happening on the Evolution front.

@tera, you've made your reservations about adding this feature clear, repeatedly. I think it's best for you to just ignore this thread now. If the feature makes technical progress and comes up for review, you will have an opportunity to give more feedback then.

6 Likes

Counterexample: half of every language feature in C++ that got pushed into the library. Move semantics, discriminated unions, SFINAE (concepts mostly fixed this one), s̴͔̮̆̄ț̴̛̝d̴̨͓͆́:̵̖̭̒:̶̖̀͛ḯ̴̘́n̸̙̐i̸̜̤͘͝t̴̪̂i̵̛̭͒a̸̻̐̾ḷ̴̩͂ỉ̵̧z̸̩̱̓é̵̮͘r̵͇̖̈́_̷̖̟͛l̶͇̀͜i̷̮̗̿̉ṣ̴̿̋ṭ̴́͑ etc.

I'm going to say something that isn't particularly relevant to the discussion occurring on this thread. Thus, we should make a new thread under Development to discuss it further. It would be helpful to plan for its implementation now instead of waiting until this SE proposal is implemented into release toolchains.

Pitch for a new thread

Making Array, Optional, Float, etc. differentiable is not part of this SE proposal, but I imagine it will come in a subsequent one. There is still a lot to decide, such as what convention we'll go by for differentiating functions that have jump discontinuities. I discussed the idea of pretending we're differentiating the raw assembly instructions with @scanon. As for Array, there are many more collection types that could be differentiated. There has been progress on a DifferentiableCollection protocol, which might be possible to merge if its blocking compiler crashers have been fixed. There are also many functions of floating-point types that do not have derivatives, and a vulnerability in their test suite that allows manual derivatives to not propagate the pullback. One more thing to discuss is the unofficial Differentiation Swift package, and its role in allowing differentiation of the Stdlib before a release toolchain officially supports that.

With my current skill set, I'm much more inclined to working on the standard library than helping push ABI stability forward. So I would be very interested in having another thread where we could exclusively discuss differentiation of standard library types. Is this discussion viable at the moment?

I don’t know if there’s a new thread now, but I just wanted everyone to know what @philipturner already read elsewhere:

There might actually be a way to move forward with ML/DL without a special autodiff toolchain. I‘m not sure about the performance of my more high level solution, but I‘m going to assess that.

Regarding apples to apples comparisons, I would hope that a good library in a good language can identify the hardware it needs and then be written against that hardware. If we’re going to assess the performance of a library, we should be using the hardware that the leaders of the field are using. Haven’t worked with swift outside the apple ecosystem a lot, don’t know how well that works by now, but I think this will be a requirement here.

It's been a little while, but to follow up on John's suggestion I've created a separate Development thread in an attempt to aggregate information about ongoing implementation work for differentiable Swift. Hopefully that will help to keep the discussions here focused on the pitch itself.

7 Likes