Creating lenses with Swift macros (usecase for arbitrary names at the global scope)

Evolution Pitches
21

@ExFalsoQuodlibet So just to be clear, you're saying this code:

let prependWorkersHomeStreet =
  \User.Lenses.workplace.workers.traverse.home.street %~ { "home " + $0 }

// some function where I'm describing state transitions:
modify(\State.Lenses.user %~ prependWorkersHomeStreet)

would be more ergonomic and better represented by the vanilla Swift:

extension User {
  mutating func prependWorkersHomeStreet() {
    for index in workplace.workers.indices {
      workplace.workers[index].home.street = "home \(workplace.workers[index].home.street)"
    }
  }
}

// some function where I'm describing state transitions:
modify { state in
  state.user.prependWorkersHomeStreet()
}

I see a few issues with this:

  1. We've reverted to C-style programming of imperatively tracking indices. This is the exact thing FP languages have been generally avoiding since the 90s, and even now essentially all languages, FP or not, prefer the use of HOFs over manual index tracking.
  2. It's 261 characters instead of 157 (66% increase) and 10 lines instead of 3.
  3. Now when I want to call my modifications, I need to pass the argument in with &, and the binding now needs to be a var ready for mutation. If I didn't want to mutate the original (I generally don't) or if it's a let (it generally is), I need to create a temporary var to hold this mutation, which won't compose well with function application. You can't just call:
someFn(modifications.map { $0(state) })

you need to instead:

var states: [State] = []
for m in modifications {
  var s = state
  m(&s)
  states.append(s)
}
someFn(states)

which feels like writing Java code from 2005 (if Java had first class functions). To be fair to the imperative code, it's better suited for reductions than mapping, and that is a common enough usecase too:

someFn <| modifications.reduce(state) { $1($0) }

vs

var s = state
for m in modifications {
  m(&s)
}
someFn(s)

but even in this best case for imperative code, I still prefer the functional solution (and it's still fewer characters and lines).

It's not that the code you sent is particularly complicated or long, but it is very imperative and relatively much longer. Scaling this up is part of the reason imperative code bases end up being much larger than FP ones in terms of LOC and character count.

It seems you're not sold on the idea of FP and related concepts (purity, HOFs, etc.) over imperative programming. That's an entirely orthogonal discussion to this post, and one that would take a while to hash out. I won't be swayed away from lenses by simply seeing longer, imperative alternatives though.

22

Actually, I am, and I've been for a very long time. But I adopt the principles (like purity and higher-order functions), and not necessarily a specific style of programming, based on composing functions with operators (which makes more sense in other languages, that lack features that Swift has though).

To me, this is a pure function

extension User {
  mutating func prependWorkersHomeStreet() {
    for index in workplace.workers.indices {
      workplace.workers[index].home.street = "home \(workplace.workers[index].home.street)"
    }
  }
}

the only difference is that it's written using regular Swift features.

The specific way in which the function is implemented is rather imperative, but that's just because Swift standard library lacks declarative functions for collection manipulation that are also mutating (something like mapInPlace). Those functions are very useful in general, so if I needed them, I would declare them in a generic way, and they would work as compositional tools in my codebase, for example:

extension MutableCollection {
  mutating func modifyEach(_ transform: (inout Element) -> Void) {
    for index in indices {
      transform(&self[index])
    }
  }
}

extension RangeReplaceableCollection {
  mutating func prepend(contentsOf toPrepend: some Collection<Element>) {
    insert(contentsOf: toPrepend, at: startIndex)
  }
}

extension User {
  mutating func prependWorkersHomeStreet() {
    workplace.workers.modifyEach {
      $0.home.street.prepend(contentsOf: "home ")
    }
  }
}

modifyEach still uses indices, but it's hardly "index tracking": it's very small and self contained. Also, if we ever get for inout, which would be a very natural extension of the language, it could be written in a even simpler way, like

extension MutableCollection {
  mutating func modifyEach(_ transform: (inout Element) -> Void) {
    for inout element in self {
      transform(&element)
    }
  }
}

to the point that there would be probably no point anymore in declaring modifyEach in the first place.

Now, having written for years code like this (not identical but very similar, and in the same spirit)

let prependWorkersHomeStreet =
  \User.Lenses.workplace.workers.traverse.home.street %~ { "home " + $0 }

// some function where I'm describing state transitions:
modify(\State.Lenses.user %~ prependWorkersHomeStreet)

I can tell you that, personally, I find my version a lot clearer and more readable, and much easier to both digest for people that are just learning the language, and to introduce in new teams, without lacking in expressive power.

Declarative programming is not really about "what" vs "how" (there's a "what" and a "how" at all levels of abstraction): it's about denotational semantics, which is still achieved by composing (pure) transformations on data structures by using regular Swift tools.

I think the strong distinction that you're making between coding styles is not particularly relevant here: for example, no one should write this code

var s = state
for m in modifications {
  m(&s)
}
someFn(s)

because Swift has already reduce(into:)

someFn(modifications.reduce(into: state) { state, modify in
  modify(&state)
})

I think the imperative Swift code that you're showing is not good, and not the code I would use for a comparison.

mutating functions (named or anonymous) are very convenient to implement, but when used in a context that requires a returned value (for example, the closure passed to map) they end up needing boilerplate (the var m_x = x boilerplate that you've shown). But this can be also solved with another natural extension to the language, the with function, so one can write

workers.map {
  $0.with {
    $0.prependWorkersHomeStreet()
  }
}

which is still compositional and denotational, but more in line, to me, with the rest of the language.

6 Likes
23

FYI, for (unfortunate) technical reasons, in Swift this really ought to be written:

extension MutableCollection {
  mutating func modifyEach(_ transform: (inout Element) -> Void) {
    var index = startIndex
    while index != endIndex {
      transform(&self[index])
      formIndex(after: &index)
    }
  }
}

I wish this weren’t the case, but as things stand the version that loops over indices can, for certain collection types, make an unwanted copy of the entire collection.

This occurs for collections where indices holds a reference to self, which means self is not uniquely referenced within the body of the loop. The first modification then triggers a copy.

As an example, the Dictionary.Values type from the standard library is susceptible to this issue:

myDictionary.values.modifyEach{ $0 += 1 }
12 Likes
24

thanks a lot, I wasn't aware of this but it's great to know :+1:

2 Likes
25

I can't combine two keypath subscripts:

struct Address { var street = "" }
struct Worker { var home = Address() }
struct Workplace { var workers: [Worker] = [] }
struct User { var workplace = Workplace() }

var user = User()

var wp: Workplace = user[keyPath: \.workplace]  // ✅
var wks: [Worker] = wp[keyPath: \.workers]      // ✅
var wks3: [Worker] = user[keyPath: \.workplace.workers] // ✅
var wks2: [Worker] = user[keyPath: \.workplace][keyPath: \.workers] // 🛑
// Error: Command SwiftCompile failed with a nonzero exit code

And compiler doesn't tell why. Looks like a bug.

26

nit:

{
    $0 = "home \($0)"
} (&workplace.workers[index].home.street)

to avoid repeating access

27

My version with the mentioned "modifyEach" and "prepend" (and lets changed to vars):

user.workplace.workers.modifyEach { $0.home.street.prepend("home ") }

Quite concise.

28

Here's the generally accepted definition of pure functions :

Even if the argument here is that structs are value types and not reference types, so there's no variation in mutable reference arguments only mutable value arguments (e.g this doesn't propagate to other bindings because creating an assignment to a value type has copy semantics instead of reference sharing semantics), it still seems to unarguably be a side effect.

an operation, function or expression is said to have a side effect if it modifies some state variable value(s) outside its local environment, which is to say if it has any observable effect other than its primary effect of returning a value to the invoker of the operation

Put another way, the function isn't referentially transparent. Even though prependWorkersHomeStreet() returns nothing, you can't replace the call to the function with nothing and end up with the same program.

I have tested it in a repl just to double check that I'm not severely misunderstanding what this does:

print(user)
let _: () = user.prependWorkersHomeStreet()
print(user)

the first and third line print different values, and removing the second line (an expression of type ()), changes the value of the program. Thus, this function emits a side effect and is not pure.

1 Like
29

prependWorkersHomeStreet is not pure per se, but functions that use prependWorkersHomeStreet can be pure, as long as User is a value type, since its side-effects are provably localized. This wouldn't be the case in languages without value types, since mutations to User can't in general case be proven to be localized for reference types.

I.e. for an outside observer

func prependHome(_ user: User) -> User {
  User(
    // create a new `User` value here from scratch
    // with the `user` argument
  )
}

and

func prependHome(_ user: User) -> User {
  var user = user
  user.prependWorkersHomeStreet()
  return user
}

are equivalent. In fact, if that's a public function you import from a different module, you may not know which of these two implementations are used, and as long as User is a value type, you won't care.

Treat var, inout, and mutating in value types as "syntax sugar" for Haskell's State monad. Creating an inout or var binding introduces new stateful scope, and you'll have this state evaluated to a result by the time this var binding goes out of scope. mutating func methods allow easy manipulation of this state without creating unnecessary copies, but you could just as well avoid mutating methods and var properties, and as illustrated by numerous posts in this thread that makes it much more clunky.

7 Likes
30

inout and mutating can be "desugared" even more simply into an in and out parameter without involving a state monad. (inout T) -> () and (T) -> T are isomorphic. If we were to introduce a "pure" concept in Swift, it would almost certainly allow the ability to modify local state and inout parameters, because the "side effects" are fully self-contained and can't affect program execution outside of those parameters. You can write:

var a = someValue
pureModify(&a)

var b = someValue
pureModify(&b)

and be sure that a and b have equivalent values.

7 Likes
31

Thanks, that's exactly what I was trying to illustrate. IIUC, in Haskell it would be expressed as

pureModify :: State T T

It would be hard to argue that this function is not pure, unless someone sneaked unsafePerformIO into its implementation.

In other words, State explicitly surfaces that effect to the user and allows it to be localized. mutating, inout, and var do the same for value types in Swift.

32

I think a closer transliteration in GHC would use linear types to represent the in-place state transition directly, T %1-> T, although even simply T -> T would suffice without involving State.

2 Likes
33

Sometimes you can externally observe the difference. Example:

struct User: Hashable {
    var name: String
    let age: Int
    let other = UUID().uuidString
    
    static func == (lhs: Self, rhs: Self) -> Bool {
        lhs.name == rhs.name && lhs.age == rhs.age
    }
    func hash(into hasher: inout Hasher) {
        hasher.combine(name)
        hasher.combine(age)
    }
}

let a = User(name: "one", age: 1)
let c = User(name: a.name + "xxx", age: a.age)
var b = a
b.name += "xxx"
precondition(b == c)             // ✅
precondition(b.other == c.other) // 💣
34

In that case UUID() isn't "pure" by either the traditional referentially-transparent definition or the value semantics definition, since you get a new UUID every time you construct one.

6 Likes
35

In what way has the pure functional code with lenses been more clunky than the non-pure imperative code that uses native swift var/inout/mutating? The functional code has used exclusively pure functions, where the non-functional code sometimes constructs pure functions out of non-pure functions, and the functional code has been shorter.

You could make the argument that the functional/lens code is not idiomatic, and that's a fair argument. It's idiomatic in different ecosystems, but by design Swift likes to handle a lot at the language level where other languages have abstractions that allow library authors to handle a lot (which introduces problems in other areas, async/throws/etc. ends up being leaking through abstractions often, where some approach like free monads in Haskell allow total abstraction over all operations and effects, where dependency inversion in Swift just allows abstraction over operations).

However the argument that it's somehow clunkier would need further clarification for something that is shorter and uses exclusively pure functions (and I hope we can all agree on the value of using pure functions). I get there are different style preferences, but I happen to like lenses and find them very readable. You can even find lens libraries that use named functions over operators like Monocle in Scala (which is also something I plan on providing in my implementation in Swift, and already have some aliases for).

I agree they're isomorphic, but they still have different syntax and semantics that lends itself to different kinds of programming. The former allows for easier construction of non-pure functions and the propagation of side effects. You can of course achieve this with the latter by assigning the result to a variable binding, but generally speaking it lends itself to construction of pure functions.

It's the same reason Haskell has both functions and Reader even though they're isomorphic. The latter often has better semantics for implicit dependency passing. While you can achieve this with the former, you don't have nicely named functions like ask or local for retrieving those dependencies in some context, nor do you generally construct functions via monad comprehensions to implicitly wire context through.

Put more generally, being isomorphic doesn't mean that they should be treated the same way, which I'm sure is something the Swift team agrees with (otherwise we wouldn't have both constructions).

36

Ah, same without UUID:

struct User: Hashable {
    var name: String
    let age: Int
    let other: String
    
    init(name: String, age: Int) {
        self.name = name
        self.age = age
        other = name + "\(age)"
    }
    
    static func == (lhs: Self, rhs: Self) -> Bool {
        lhs.name == rhs.name && lhs.age == rhs.age
    }
    func hash(into hasher: inout Hasher) {
        hasher.combine(name)
        hasher.combine(age)
    }
}

let a = User(name: "one", age: 1)
let c = User(name: a.name + "xxx", age: a.age)
var b = a
b.name += "xxx"
precondition(b == c)              // ✅
precondition(b.other == c.other)  // 💣
37

Yes, that's the argument I'm personally making. Overloaded operators for non-numeric code are not idiomatic in Swift, and without those I don't think you'll get the same brevity in lenses sample code. Additionally, curried functions and point-free programming are not idiomatic in Swift either, at least when comparing to how these are pervasive in Haskell.

2 Likes
38

Your initializer in that case still isn't a true memberwise initializer, since it initializes other in a parameter-dependent way, so the equivalence we're talking about doesn't apply.

2 Likes
39

All I am saying is that sometimes the difference is externally observable...

Same example with "normal" memberwise initialiser 
struct User: Hashable {
    var name: String {
        didSet { other += "+++" }
    }
    var age: Int
    var other: String
    
    static func == (lhs: Self, rhs: Self) -> Bool {
        lhs.name == rhs.name && lhs.age == rhs.age
    }
    func hash(into hasher: inout Hasher) {
        hasher.combine(name)
        hasher.combine(age)
    }
}

let a = User(name: "one", age: 1, other: "other")
let c = User(name: a.name + "xxx", age: a.age, other: "other")
var b = a
b.name += "xxx"
precondition(b == c)
precondition(b == c)              // ✅
precondition(b.other == c.other)  // 💣
40

Key paths and case paths are just another name for (simple) lenses and prisms, and so they compose the same. In the lens hierarchy it's called an affine traversal, and Swift models this kind of getting/setting via optional-chaining.

We've called it an "optional path" before:

struct WritableOptionalPath<Root, Value> {
  let get: (Root) -> Value?
  let set: (inout Root, Value) -> Void
}

The big issue is that while key paths and case paths theoretically compose this way, we would need to introduce explicit conversions to make it work, which isn't the most ergonomic. We've found that these kinds of compositions fit more cleanly in Swift by structuring the composition in the operations themselves rather than composing the optic.

.ifLet(\.case) { // case key path
  Scope(state: \.child) { // key path
    …
  }
}

// instead of:
.ifLet(
  WritableOptionalPath(\.case).appending(
    path: WritableOptionalPath(\.child)
  )
) {
  …
}

Of course, ideally Swift's native optics story evolves to the point of shipping first-class case key paths and writable optional key paths. Then you would be free to do the following:

.ifLet(\.case?.child) {
  …
}

Added:

Many years ago Joe showed that traversals are possible in some capacity via key path subscripts:

5 Likes