Creating lenses with Swift macros (usecase for arbitrary names at the global scope)

I'm writing a library that is creating an implementation of lenses (inspired by Haskell). Currently, I've got an implementation of relatively simple lenses. You can generate lenses like so:

@makeLenses
struct Address {
  let street: String
  let city: String
  let state: String
  let zipCode: Int
}

@makeLenses
struct User {
  let name: String
  let age: Int
  let address: Address
}

this will create static vars inside the structs that match the names of the fields. Ideally I would actually create the lenses at the same level as the struct instead of inside the struct, but this doesn't always work, you can't introduce arbitrary names in a macro at the global scope.

I currently have 3 variants of @makeLenses.

  • @makeLenses - this will create static vars inside the struct.
  • @makeLensesPeer - this will create static vars at the same level.
  • @makeLensesPeerNonStatic - this will create normal vars at the same level. Not really worth using with the current state of Swift Macros, which is why the name is unwieldy.

What the above code allows is code like so:

let changeStreet: (User) -> User = 
  User.address ~ Address.street .~ "new street name"

or if you are inside a struct and were able to do @makeLensesPeer:

let changeStreet: (User) -> User =
  address ~ street .~ "new street name"

as opposed to:

func changeStreet(_ user: User) -> User {
  User(
    name: user.name,
    age: user.age,
    address: Address(
      street: "new street name",
      city: user.address.city,
      state: user.address.state,
      zipCode: user.address.zipCode
    )
  )
}

My main issue with this currently is if you're adding lenses to another struct inside of some struct (say a State struct inside of some Feature), then what you can easily end up with is:

let changeStreet: (UserFeature.State) -> UserFeature.State =
  UserFeature.State.address ~ AddressFeature.State.street .~ "new street name"

Which isn't nice. Assuming we aren't getting the arbitrary names at the global scope limitation lifted, my next best solution is to have a type Lens with an alias L, and then have a freestanding macro #makeLenses create an extension to Lens with static vars (with accessors), so I could get this code:

// at global scope
#makeLenses(UserFeature.State.self)
#makeLenses(AddressFeature.State.self)

// anywhere in the program
let changeStreet: (User) -> User =
  L.address ~ L.street .~ "new street name"

Is this feasible given the current implementation of macros? If not, does anyone have alternative suggestions to easing use of lenses across the program?

2 Likes

IIUC Swift's KeyPath class hierarchy from the standard library matches the FP lenses concept quite closely, if not exactly. Is there a reason to generate your own lenses boilerplate code instead of relying on keypaths? They're just as composable as lenses that you'd be defining manually, and as long as you use structs instead of classes, functions you create with them can stay pure, even if properties of these structs are defined with var and not let:

struct Address {
  var street: String
  var city: String
  var state: String
  var zipCode: Int
}

struct User {
  var name: String
  var age: Int
  var address: Address
}

let changeStreet = \User.address.street

let changeStreetFeature = \UserFeature.State.address.street

var user = User(/* initial values */)
var userFeature = UserFeature.State(/* initial values */)

user[keyPath: changeStreet] = "new street name"
userFeature[keyPath: changeStreetFeature] = "new street name"

As a side note, var properties in value types give you "syntax sugar" that roughly corresponds to Haskell's State monad. Mutable properties in structs stay immutable if the struct itself is bound as a constant let value. And you can copy that let constant into a locally mutable value by rebinding it with var, then implicitly convert it back to immutable by returning it from function. That is, this code you've written

is equivalent to

// this function is pure, as long as `User` is a `struct`
// or `enum` preserving value semantics
func changeStreet(_ user: User) -> User {
  var user = user // copy immutable into mutable value
  user.address.street = "new street name" // modify locally
  return user // return back as immutable again
}

you don't even need keypaths to write the latter.

9 Likes

In my experience, lenses only make sense in Swift if you're using structs with let properties while declaring (explicitly or implicitly) a memberwise initializer, which you should not do: if there's a memberwise initializer, properties should be var which make the structs very easy to manipulate.

Even if you're thinking about using lenses to represent and store the concept of mutation for a struct, there's simpler ways of doing that, that involve keypaths and closures. For example:

struct Address {
  var street: String
  var city: String
  var state: String
  var zipCode: Int
}

struct User {
  var name: String
  var age: Int
  var address: Address
}

let changeStreet = { newStreet in
  { (user: inout User) in
    user.address.street = newStreet
  }
}

var user = User(name: "Yello", age: 42, address: .init(
  street: "Old Street",
  city: "Old Town",
  state: "Freedonia",
  zipCode: 12345
))

let update = changeStreet("New Street")
update(&user)

I had used keypaths previously, but they fall short in a number of ways.

I'm not convinced that local reasoning is identical with vars and lets. Having an immutable struct of data signals that it'll literally never mutate, and that it can only change during copy. Making it mutable requires that you scan the entire context and look for & or simple mutation to verify that it's never mutated. If you're not convinced of this point, do you literally never use let?

They can match simple operations (e.g .~), but more advanced operations take a lot more code. Take the following code snippet:

// if we're inside a struct so we can use `makeLensesPeer`
// or if the global arbitrary name restriction is lifted

@makeLensesPeer struct Address {
  let street: String
  let city: String
  let state: String
  let zipCode: Int
}

@makeLensesPeer struct Workplace {
  let workers: [Worker]
  let name: String
  let address: Address
}

@makeLensesPeer struct Worker {
  let name: String
  let home: Address
}

@makeLensesPeer struct User {
  let name: String
  let age: Int
  let home: Address
  let workplace: Workplace
}

then we could write:

let prependWorkersHomeStreet: (User) -> User =
  workplace ~ workers ~ traverse() ~ home ~ street %~ { "home " + $0 }

without lenses we would instead change the lets to vars, and then write:

func prependWorkersHomeStreet(_ user: User) -> User {
  var user = user
  user.workplace.workers = user.workplace.workers.map { worker in
    var worker = worker
    worker.home.street = "home " + worker.home.street
    return worker
  }
  return user
}

important context to this too, this is all happening in a framework where it's common to pass descriptions of state changes, e.g (State) -> State, so you might see code like:

case .changeName(let newName):
  modify(name .~ newName)
case .prependStreet(let prefix): 
  modify(home ~ street %~ { prefix + $0 })
case .incrementAge: 
  modify(age +~ 1)

without lenses (and requiring vars on all the fields, which is a negative), these types of modifications would be described as:

case .changeName(let newName): 
  modify { user in
    var user = user
    user.name = newName
    return user
  }
case .prependStreet(let prefix): 
  modify { user in
    var user = user
    user.home.street = prefix + user.home.street
    return user
  }
case .incrementAge: 
  modify { user in
    var user = user
    user.age += 1
    return user
  }

I had previously built a version that used keypaths, but it wasn't as powerful. Technically writablekeypaths can match lenses so long as you are fine dropping lets on all your fields (I am not), but they don't match other optics like prisms or traversals. I had also used the casepaths library for matching prisms, but still there was nothing to match traversals. Plus the syntax for keypaths and casepaths being inverted meant a cognitive load that isn't present with lenses/prisms. Instead of having to manage modify(\User.name .~ newName) in some contexts and then scoped(/SomeEnum.someCase) in other contexts, lenses/prisms allow modify(User.name .~ newName) and scoped(SomeEnum.someCase).

it'd be hard to convince me that the native solution here is objectively better or at least just as good as the lenses, and it also comes with the downside of requiring vars on all the fields, which opens up mutation when I would rather have lets (again I ask do you literally never use lets? If you do, why are you okay dropping it even if you never plan on mutating the fields?)

var properties of structs are only mutable when the struct value as a whole is mutable, so a let struct will still give you the immutability guarantee for a value that you want, while still allowing you to make copies with adjusted values. Because structs have value semantics, there is no meaningful difference between having a memberwise initializer and having the members be individually mutable, because these are equivalent:

let a = Address(...)
let b = Address(street: "123 New Street", city: a.city, state: a.state, zipCode: a.zipCode)
let a = Address(...)
let b = {
  var tmp = a
  tmp.street = "123 New Street"
  return tmp
}()

In both cases, a and b are completely immutable, independent values, with only one field's value difference between them. Using let for struct properties can make sense when there are invariants that ought to hold between fields that the initializers enforce, but for simple collections of fields like in your examples, where memberwise initialization is already part of the API of the type, you'll have an easier and more efficient time letting the fields be vars without sacrificing any control over mutability. Then you can implement those lens operators using the builtin WritableKeyPath type. (There are nonetheless still yet more advanced things that Haskell's lens/prism/traversal types can do which KeyPath can't, and it can be a worthwhile experiment with macros to see whether language-integrated functionality can be supplanted by libraries now. I think a Swift-native lens solution should still embrace value semantics rather than uncritically follow the immutability strictures of functional languages.)

14 Likes

Imagine you have a function that is 80 lines long (bad practice generally, but for the sake of argument). In that function, you see on line 15:

let someValue = someFn(user)

then on line 60 you see:

let myValue = someFn(user)

if all your fields/variables in your entire program are immutable (e.g all structs are defined with lets and not vars), then you know these two are identical, and you can remove the latter. You don't need to do further inspection. In Swift, this isn't guaranteed by the language, and so either it's guaranteed by the project (as a practice you never allow var no matter what), or you have to do further inspection (you've lost some local reasoning).

Of course this is better than Python, or even Java, but the guarantees aren't as strong as say Haskell. There is value in global immutability, and being forced to lose that to allow changing during copy, when lenses/optics fill the exact same role in a more concise and powerful way seems pointless.

But in Swift, if someValue and myValue are a struct type, then whether the struct's fields are themselves var or let is irrelevant to the guarantee you're looking for. someValue is a let so its own field values can't change. You would have to make a mutable copy to be able to modify its var fields if it has any, but that can't affect the value of someValue.

6 Likes

Even though someValue and myValue are lets, they could technically be different values because user could have changed between line 15 and 60, and you'd have to do inspection to verify that user is either a let, or that it hasn't been mutated in those 45 lines. In Haskell if you see some redundancy like this, you know immediately you can remove it, always, without any further thought (shadowing being the exception, but by default it's a warning and you can configure it to be an error in a project, which I think is good to do).

Even though someValue and myValue are lets, they could technically be different values because user could have changed between line 15 and 60, and you'd have to do inspection to verify that user is either a let , or that it hasn't been mutated in those 45 lines.

Again, that's still only a concern if the variable user is itself mutable, not whether user's struct type contains var properties. You can use immutable bindings everywhere outwardly while still taking advantage of value semantics and key paths to give you composable update operations on value types.

8 Likes

To be even more clear, the point is that if user is a var at all then it can still change.

func someReallyLongFunction() {
    var user = User(...)
    let someValue = pureFunction(user)
    ...
    user = User( ... completely different values ... )
    ...
    let otherValue = pureFunction(user) // probably not the same as someValue
}

Whereas if user is a let to begin with, this can't happen. Even if User has var properties.

7 Likes

Yes, and you don't know if user is a var unless you've looked. This is local reasoning that's been lost. If you only ever use lets then you know from the outset that it's not mutable.

Unless you want to change something during a copy, then you have to use a var binding. Having the rule "vars aren't allowed ever" is much easier than "vars aren't allowed ever except as fields for structs and if you're specifically in a context where you're constructing a temporary reference to an existing variable to allow change during copying"

I do understand, afaict, how let and var work. People have been explaining it to me, maybe thinking I don't already know, but I haven't actually learned anything from these explanations, because I know what you can achieve with lets and vars.

The thing nobody has addressed so far is why I would want to drop lenses for this worse solution. The argument everyone is making is "you don't really lose that much by varing all your fields", and then I ask why do you ever use let for a field declaration? Do you never? Do you literally var every field in your program? I'd imagine very few swift developers do this, because everyone understands the value of immutability and local reasoning.

Assuming you do let a field ever, then that's a tacit admission that being required to use vars to enable some feature is a negative. You might think it's not a very big negative, maybe you think it's almost negligible, but if you thought it was actually negligible, then you would never let a field, and you would always var fields for both consistency and to enable more functionality.

I find it interesting that this line of reasoning can be simultaneously interpreted as an argument for all properties to be var, and also an argument for all properties to be let:

“The mutability of user depends solely on whether user itself is declared var or let. It does not depend on how the member properties of user are declared. Therefore the member properties of user should be declared var because that is more convenient. There is nothing special about user, so it follows that in general the properties of structs should be var.”

But also:

“The mutability of user depends solely on whether user itself is declared var or let. Often, user will be a property of some larger struct, and the function we are writing will be a method on that struct. To ensure immutability of user in that situation, user must be declared let. There is nothing special about user, so it follows that in general the properties of structs should be let.”

• • •

…of course, within a method on a struct, the properties of that struct are immutable even if they are declared var. They are only mutable within a mutating method. So that is why the second interpretation fails. There really is no reason to declare most struct properties as let. They should pretty much always be var.

1 Like

Honestly, the only time I'd ever add a let field to a struct would be to add a ID to it, which usually means I'm implementing some form of reference type (and I'd probably use a move only struct for that these days). Otherwise I really do never let a struct field. Whether the variable I have is a let or a var is all the local reasoning I need, and is in fact the entire reason mutability works the way it does in Swift. All the benefits of immutable types, none of the boilerplate.

Now, for classes let fields are great, but I don't write many of those.

6 Likes

I do somewhat regret that we use "let" and "var" for struct fields the same way we do for variables and properties elsewhere, because var-ing a struct field doesn't introduce nonlocal reasoning in the same way that var-ing a variable or class property does. The main effect is really to say "this field's value is independent from the other fields", and my own advice is generally to say that most struct properties should be var unless there are cross-field invariants that mean it doesn't make sense to independently update that field. That's also an interesting property for even a purely immutable lens library to take into account, since per-field lens operations can potentially break cross-field invariants too.

14 Likes

To be pedantic, you won't know this with struct properties either. Any of struct properties could be a class instance not behaving as a value type, breaking your assumptions. But that would be violation of the contract if the struct pretended to be a value type in the first place. In the same way, one could say you have to audit all Haskell code to check that your pure functions don't make calls to unsafePerformIO under the hood.

Based on this, I recommend focusing not on let vs var, but on reference vs value semantics instead. That's what gives you localized mutability, which allows functions to stay pure, if needed. let on struct properties only adds unnecessary boilerplate most of the time at that point.

Mutability and local reasoning are not mutually exclusive when mutability itself is local. IIUC this is what Haskell's State monad shows, where in a pure functional language you can localize mutation to a closure passed to an evalState call. In Swift, as long as you operate on value types, you get this local mutability on the language level instead of a monad library level.

Even the inout parameter modifier is defined not in terms of reference semantics, but in terms of mutation local to a var binding passed to such function call, allowing value semantics to be preserved even when calls to inout functions are present:

In-out parameters are passed as follows:

  1. When the function is called, the value of the argument is copied.
  2. In the body of the function, the copy is modified.
  3. When the function returns, the copy’s value is assigned to the original argument.

This behavior is known as copy-in copy-out or call by value result.

5 Likes

@Tylerwbrown A lot of people have given you reasons for why you shouldn't do this, and I 100% agree with them, but there is a way to accomplish what you want with some tricks.

Before demonstrating these tricks I want to make it abundantly clear that I do not think any of this is actually necessary, and key paths should be used for this. I even gave a popular talk on lenses in Swift 8 years ago (almost to the day!), but that pre-dated native Swift key paths by 2 years and once native key paths were available I converted all code to use them.

Key paths are far more ergonomic, more powerful, and adopting a "lets everywhere, vars nowhere" mantra is only working against the language and will lead to pain down the road. Value types give you all the safety and locality you need, even with vars, and even adopting the mantra does not change the fact that all other code you interact with happily uses vars (such as the standard library).

But with that caveat aside, you can adopt the approach we took to develop our CasePaths library to accomplish what you want. The crux of the technique is that we want to leverage native key path syntax for something that is not natively supported by Swift, such as key paths for cases of an enum (or "prisms" as they are known in functional languages). We of course really wish Swift would have native support for enum key paths, but until then we were able to brute force it in with macros.

Here is how it goes, slightly adapted for "lenses" instead of "prisms". The trick is to house the lenses inside the type that you want to derive lenses for, similar to what your @makeLenses does, but instead of static vars it is a new nested type with vars. Here is a small amount of library code you have to maintain to abstract over this concept and make it composable:

protocol Lensable {
  associatedtype AllLenses
  static var allLenses: AllLenses { get }
}

typealias Lens<Root: Lensable, Value> = KeyPath<Root.AllLenses, AnyLens<Root, Value>>

@dynamicMemberLookup
struct AnyLens<Root, Value> {
  let get: (Root) -> Value
  let set: (Root, Value) -> Root

  subscript<AppendedValue>(
    dynamicMember keyPath: KeyPath<Value.AllLenses, AnyLens<Value, AppendedValue>>
  )
  -> AnyLens<Root, AppendedValue>
  where Value: Lensable
  {
    AnyLens<Root, AppendedValue>(
      get: { Value.allLenses[keyPath: keyPath].get(self.get($0)) },
      set: { self.set($0, Value.allLenses[keyPath: keyPath].set(self.get($0), $1)) }
    )
  }
}

With that you can define the AllLenses inner type inside your models holding onto the lenses, and you can make the models conform to Lensable (easily accomplished with a @Lensable macro):

struct User {
  let id: Int
  let address: Address

  // Macro generates this
  static let allLenses = AllLenses()
  struct AllLenses {
    let id = AnyLens(get: \.id, set: { User(id: $1, address: $0.address) })
    let address = AnyLens(get: \.address, set: { User(id: $0.id, address: $1) })
  }
}
// Macro generates this
extension User: Lensable {}

struct Address {
  let city: String
  let state: String

  // Macro generates this
  static let allLenses = AllLenses()
  struct AllLenses {
    let city = AnyLens(get: \.city, set: { Address(city: $1, state: $0.state) })
    let state = AnyLens(get: \.state, set: { Address(city: $0.city, state: $1) })
  }
}
// Macro generates this
extension Address: Lensable {}

And with that done you immediately get to use native Swift key path syntax to derive lenses:

let userAddressLens: Lens<User, Address> = \User.AllLenses.address

And then key path composition corresponds to lens composition:

let userCityLens: Lens<User, String> = \User.AllLenses.address.city

That's right. userCityLens is a proper lens focusing from User all the way down to the String of the city, albeit expressed in key path syntax. All model fields are still lets, yet you can make a mutation to a user's city:

let user = User(id: 42, address: Address(city: "Brooklyn", state: "NY"))

// Relocate user from Brooklyn to Queens
let relocatedUser = User.allLenses[keyPath: userCityLens].set(user, "Queens")

This works even though user and all of its fields are completely immutable. It's a little verbose, but you can define your .~ operator in terms of Lens so that you can do things like this:

let user = User(id: 42, address: Address(city: "Brooklyn", state: "NY"))

// Relocate user from Brooklyn to Queens
let relocatedUser = user
  |> userCityLens .~ "Queens"

…but I will leave the implementation of .~ as an exercise for the reader. :slightly_smiling_face:


So, this is all possible, but I will iterate again that I personally do not think this buys you anything over Swift's value semantics and key paths, and in fact only hurts the quality of your code and the ability to take full advantage of what the Swift language has to offer.

But, the main reason I wanted to post this solution is just to make it more widely known how one can leverage Swift key path syntax for things that are not exactly key paths, and how key path composition can be naturally transported to other kinds of composition[1]. It is incredibly handy, and it's what allowed us to make "case key paths" seem as native as key paths, though we still do hope for a day of truly native case paths. :slightly_smiling_face:


  1. Technically you need to do a bit more work to make all kinds of key path composition to work, but you can see our CasePaths library for the full set of tricks. ↩︎

31 Likes

Of course I do. I do when I need to maintain some invariant, for example if a struct can only be initialized from some other value, such that the properties of the former must be in some strict relationship between themselves, depending on the properties of the latter. But this means that the original struct doesn't declare a memberwise initializer: if it does (with the same visibility of the properties), then it can always be reconstructed from another instance of the same struct, making let properties pointless.

In fact, this makes let a powerful strategic tool for structs, that's completely lost if you declare (again, unnecessarily) all properties as let. These questions of yours

plus your arguments

and

tell me that you probably don't fully understand value semantics in Swift, and how it can enforce invariants, and you should probably take another look.

The language 100% guarantees that, even if a struct has var properties, instances assigned to let variables cannot "mutate", preserving local reasoning, and that's actually one of the most important Swift features, that sadly many people don't fully get.

As a side note: like @mbrandonw, many years ago I was also into Swift optics (this old article of mine is a decent proof) but I realized over the years that I wasn't really using Swift as it's supposed to be used, probably because I didn't fully understand some aspects of the language.

6 Likes

@mbrandonw Thanks for the write up, there's a lot of info here and I'll take some time to dissect it.

So far I'd say 80->90% of this discussion has been around value semantics and whether or not keypaths can achieve what lenses can achieve. There has been the repeated assertion that keypaths/native Swift can match or exceed the ergonomics of lenses, without much discussion surrounding that.

and to be fair, there has been some support for the opposite position (that lenses can actually exceed keypaths in ergonomics):

which is the camp I'm also in. I'll use a code example I wrote earlier highlighting the differences between a native approach and a lens approach, and I'm curious if the people who advocate for the universal usage of native Swift solutions over lenses think the native code snippet is superior enough that the idea of making/using lenses should be entirely discarded in the Swift ecosystem.

This is what I would have in Swift, assuming you have these lenses in scope (if you happen to be in a struct that defined the other structs with @makeLensesPeer):

let prependWorkersHomeStreet =
  workplace ~ workers ~ traverse() ~ home ~ street %~ { "home " + $0 }

That won't happen very often though, so more commonly you'd see:

let prependWorkersHomeStreet =
  User.workplace 
    ~ Workplace.workers 
    ~ traverse() 
    ~ Worker.home 
    ~ Address.street 
    %~ { "home " + $0 }

Using a solution like @mbrandonw laid out, it seems like it'd be possible to end up with something like this instead:

let prependWorkersHomeStreet =
  \User.Lenses.workplace.workers.traverse.home.street %~ { "home " + $0 }

not exactly sure how the composition of lenses and traversals would work in that last example, but something like that is probably doable.

Finally, here's the native, non-lens solution:

func prependWorkersHomeStreet(_ user: User) -> User {
  var user = user
  user.workplace.workers = user.workplace.workers.map { worker in
    var worker = worker
    worker.home.street = "home " + worker.home.street
    return worker
  }
  return user
}

I'm not a fan of the boilerplate required for the native solution. Specifically having to create a new binding and then return that binding, which we see a couple times above:

var x = x
// some mutation on x
return x

and having to reference what you're changing both when you're setting and getting it, which we also see a couple times above:

x.some.nested.field = change(x.some.nested.field)

both those together:

var x = x
x.some.nested.field = change(x.some.nested.field)
return x

with lenses would be:

\X.Lens.some.nested.field %~ change

For those who promote using native Swift solutions rather than lenses, do you see no advantages to the use of lenses here? Would people still claim that keypaths are universally more ergonomic and powerful?

If it's someone's position that there is value in these operators, e.g .~, %~, but that the use of the Lens type on the left hand side should be replaced with WriteableKeyPaths, then it almost would be worth returning the discussion exclusively towards value semantics and whether or not using lets on structs is a good idea, except that I actually use other optics too (e.g traversals and prisms).

Prisms can be matched by the CasePath library, but afaik there's nothing to match the general concept of traversals, and I'm also not sure that keypaths and casepaths compose as nicely as lenses and prisms (I would have to look into it more).

1 Like

Not to speak for Joe here, but I doubt he meant that lenses themselves are more powerful than what you can regularly achieve in Swift today; he mentioned "lens/prism/traversal types", and it's absolutely true that:

  • Swift doesn't natively have anything that works like prisms;
  • traversals cannot probably be represented in Swift at all in a truly generic way without emulating higher-kinded types.

here's a more concise version

extension User {
  mutating func prependWorkersHomeStreet() {
    for index in workplace.workers.indices {
      workplace.workers[index].home.street = "home \(workplace.workers[index].home.street)"
    }
  }
}

there's no native symmetric operator to .append, but it would be easy to add it, making this even clearer and simpler, something like:

extension User {
  mutating func prependWorkersHomeStreet() {
    for index in workplace.workers.indices {
      workplace.workers[index].home.street.prepend(contentsOf: "home ")
    }
  }
}

This, to me, is closer to how Swift should be written in order to compare its "style" with comparable solutions.

It's generally better, in Swift, to represent functions like these as (inout State) -> Void, which makes them immediately more ergonomic, because the calls become

modify { user in
  user.name = newName
}

I think that the way you're writing native solutions is not fully using language features, and looks like similar solutions in other languages, crudely translated into Swift.

7 Likes