Upstreaming '@differentiable' attribute and 'Differentiable' protocol

Development Compiler
1

As we get closer to being ready to upstream the differentiable programming feature to the master branch, I have a question about how to gate the feature.

There are 4 major user-visible components:

  1. @differentiable declaration attribute
  2. @differentiable function type attribute
  3. Differentiable protocol
  4. Differential operators (top-level functions in stdlib)

Option 1: Underscore all of these things.

  • Pros: Easy to do. Consistent with other experimental features.
  • Cons: We'll need to drop the underscore every time we build a Swift for TensorFlow toolchain.

Option 2: Add a frontend flag --enable-experimental-differentiable-programming. When disabled, all above features are unavailable.

  • Pros: More principled.
  • Cons: A slight bit more engineering. Idk if there's a precedent.

@tkremenek and the core team, what would you recommend?

3 Likes
2

I'll add option 3:
Option 3: Upstream all stdlib changes to a separate _Differentiation module. Importing this module turns on the feature and makes @differentiable attributes available.

  • Pros: No underscoring or compiler flags needed. The 'tensorflow' branch can use @_exported import _Differentiation to enable it by default.
  • Cons: No precedent.
3 Likes
3

I’m not sure if my opinion is worth anything here, but I’d like to say that I really am fond of option 3. Having the functionality be handled by a separate module makes it easier to include or exclude the functionality. This is extremely interesting from an embedded systems perspective where we may not have the space for an extremely large library. Splitting it up into smaller pieces makes it composable and th system can easily determine if that is needed or if not make the trade off to remove it.

I think that there is a reasonable approach to handle this in the compiler even. We could generalize the way that the standard library works with a set of well known modules that can be imported based on the flags.

6 Likes