1

@Troy_Harvey and I made a series of AutoDiff tutorials that try to provide a smooth ramp-up for a beginner to understand AutoDiff's usefulness, and also how to wield it. See also this and this.

Part 0: Why Automatic Differentiation is Awesome
Part 1: Gradient Descent
Part 2: Differentiable Swift
Part 3: Differentiable API Introduction
Part 4: Differentiable Swift API Details

20 Likes
2

I think there’s a typo in part 2, in the example with f(x) = x².

After these things:

var gradientAtThree = gradient(at: 3, of: f)
// gradientAtThree: 6.0

let stepInXDirection = -gradientAtThree
let newInput = 3 + stepInXDirection

It says:

//newOutput: 5.76

But, unless I’m missing something, newInput is -3, so newOutput would still be 9.

It looks like the output was calculated for 2.4, meaning stepInXDirection was intended to be -0.6 instead of -6.0.

1 Like
3

Indeed, thank you! I fixed it.

2 Likes
4

and i thought swift is big already!

was an interesting reading, thanks.

i remember the below experiment of mine, where i defined a custom Op type backed by enum and redefined all math operations on it. it's of course not nearly as advanced as what can be achieved with compiler support but it's still something capable of doing forward mode derivatives in symbolic and numeric forms. i wonder if this direction was explored as an alternative to consider and what are the pros and cons of such "library" approach compared to autodiff.

forward mode symbolic derivative experiment 
indirect enum Op: Hashable {
	case minus(Op)
	case add(Op, Op)
	case sub(Op, Op)
	case mul(Op, Op)
	case div(Op, Op)
	case power(Op, Op)
	case num(NumberType)
	case name(String)
	case sinus(Op)
	case cosinus(Op)
	.....
}
...
let x = Op("x")

// example1
let f = x^2
f.debug()                           // prints: x^2
let dfdx = f.derivative(x)
dfdx.debug()                        // prints: 2*x
let substituted = dfdx.substitute([x: 3])
substituted.debug()                 // prints: 2*3
let calculated = substituted.calc()
calculated.debug()                  // prints: 6

// example2
let z = 2*(3+x^3)					// prints:
z.debug()                           // 2*((3+x)^3) 🐞 FIX brackets 
z.substitute([x: 1]).debug()        // 2*((3+1)^3)
z.substitute([x: 1]).calc().debug() // 128
let dzdx = z.derivative(x).debug()  // 2*(3*((3+x)^2))
dzdx.substitute([x: 1]).debug()     // 2*(3*((3+1)^2))
dzdx.substitute([x: 1]).calc().debug()  // 96
1 Like
5

I believe they are considered as part of eDLS exploration: from OP's link,
Differentiable Programming Manifesto – Approaches to Automatic Differentiation.

2 Likes