 # AutoDiff tutorials

@Troy_Harvey and I made a series of AutoDiff tutorials that try to provide a smooth ramp-up for a beginner to understand AutoDiff's usefulness, and also how to wield it. See also this and this.

Part 0: Why Automatic Differentiation is Awesome
Part 2: Differentiable Swift
Part 3: Differentiable API Introduction
Part 4: Differentiable Swift API Details

18 Likes

I think there’s a typo in part 2, in the example with f(x) = x².

After these things:

``````var gradientAtThree = gradient(at: 3, of: f)
``````
``````let stepInXDirection = -gradientAtThree
let newInput = 3 + stepInXDirection
``````

It says:

``````//newOutput: 5.76
``````

But, unless I’m missing something, `newInput` is -3, so `newOutput` would still be 9.

It looks like the output was calculated for 2.4, meaning `stepInXDirection` was intended to be -0.6 instead of -6.0.

1 Like

Indeed, thank you! I fixed it.

1 Like

and i thought swift is big already!

i remember the below experiment of mine, where i defined a custom Op type backed by enum and redefined all math operations on it. it's of course not nearly as advanced as what can be achieved with compiler support but it's still something capable of doing forward mode derivatives in symbolic and numeric forms. i wonder if this direction was explored as an alternative to consider and what are the pros and cons of such "library" approach compared to autodiff.

forward mode symbolic derivative experiment
``````indirect enum Op: Hashable {
case minus(Op)
case sub(Op, Op)
case mul(Op, Op)
case div(Op, Op)
case power(Op, Op)
case num(NumberType)
case name(String)
case sinus(Op)
case cosinus(Op)
.....
}
...
let x = Op("x")

// example1
let f = x^2
f.debug()                           // prints: x^2
let dfdx = f.derivative(x)
dfdx.debug()                        // prints: 2*x
let substituted = dfdx.substitute([x: 3])
substituted.debug()                 // prints: 2*3
let calculated = substituted.calc()
calculated.debug()                  // prints: 6

// example2
let z = 2*(3+x^3)					// prints:
z.debug()                           // 2*((3+x)^3) 🐞 FIX brackets
z.substitute([x: 1]).debug()        // 2*((3+1)^3)
z.substitute([x: 1]).calc().debug() // 128
let dzdx = z.derivative(x).debug()  // 2*(3*((3+x)^2))
dzdx.substitute([x: 1]).debug()     // 2*(3*((3+1)^2))
dzdx.substitute([x: 1]).calc().debug()  // 96
``````
1 Like

I believe they are considered as part of eDLS exploration: from OP's link,
Differentiable Programming Manifesto – Approaches to Automatic Differentiation.

2 Likes