[Pitch] `@OptionSet` macro

[anon9791410]

@michelf

current design

To reiterate my previous post with a simpler example, there's nothing that breaks the current "design" by separating the conflated type. You just have to copy over an implementation for contains, insert, remove, and update.

enum ShippingOption: Int {
  case nextDay, secondDay, priority, standard

  struct Set: OptionSet {
    typealias Element = ShippingOption
    let rawValue: Element.RawValue
  }
}

[Jon_Shier]

@michelf

Ideally I'd just write this:

@AutoFill // would be nicer if this was unnecessary
struct ShippingOptions: OptionSet {
  static let nextDay
  static let secondDay
  static let priority
  static let standard
}

I like this, but I feel like Swift should adopt a macro policy that says if you forget to add the macro, the code either fails to compile in a way that tells you use the macro, or is valid on its own. Unfortunately there's no way to do that with static values without initial values, which rather defeats the purpose of this macro. I also rather dislike any of the transforms that veer too far from what the user has written. The original proposal meets the first rule easy, but fully synthesizing the static members of the options may be too far.

I do like the idea of replacing the OptionSet conformance with @OptionSet if you want the auto synthesis. It sets up a good pattern going forward. Another option is something like @synthesize(OptionSet) or something like that, but that doesn't seem to scale well for macros.

[michelf (Michel Fortin)]

@Jon_Shier

I like this, but I feel like Swift should adopt a macro policy that says if you forget to add the macro, the code either fails to compile in a way that tells you use the macro, or is valid on its own. Unfortunately there's no way to do that with static values without initial values, which rather defeats the purpose of this macro.

Is this really reasonable to expect this from macros when we already don't expect this from property wrappers?

import SwiftUI
struct ContentView: View {
  @Environment(\.locale) var locale  // look, no value and no type!
  ... // also, no init()
}

But I suppose we could use static var if it's the let you find objectionable.

[Jon_Shier]

This is also a bad developer experience. I see no reason why we need to use the poor precedent of these property wrappers in a more general feature like macros. It certainly doesn't justify another poor experience.

[esummers (Eric)]

@michelf

An by "redesign" I mean there is almost nothing in common between this and a normal option set declaration:

@OptionSet
struct ShippingOptions {
    private enum Options: Int {
        case nextDay, secondDay, priority, standard
    }
}

SwiftUI declares options sets similar to this with an enum. I think it makes sense because a set should have an element type.

I suggested changing it slightly from the pitch here, so you have more power over how the enum is exposed to the user. You could make the enum private, put the option set inside the enum, put the enum inside the option set, or use an enum defined elsewhere.

[Douglas_Gregor]

@esummers

@tera

Interestingly, it is reversed compared to an example in the motivation section of the pitch (not an enum inside a struct but a struct inside an enum).

I wonder if this pitch should change so it could be used in SwiftUI?

I found it interesting how we've had at least different ways in which we could describe option sets. There's the original pitch (with the nested, private Options enum), this SwiftUI pattern (with a nested Set inside a public enum), and the static-variables approach.
(There's also the struct-of-Bool-instance properties, but let's set that aside for the moment).

So, I went ahead and implemented all three. The same macro can handle all three forms by inspecting the syntax of the declaration to which it is attached.

Form #1:

@OptionSet<UInt8>
struct ShippingOptions {
  static var nextDay: ShippingOptions
  static var secondDay: ShippingOptions
  static var priority: ShippingOptions
  static var standard: ShippingOptions
}

Form #2:

@OptionSet<UInt8>
struct ShippingOptions {
  enum Options {
    case nextDay
    case secondDay
    case priority
    case standard
  }
}

Form #3:

@OptionSet<UInt8>
enum Options: UInt8 {
  case nextDay
  case secondDay
  case priority
  case standard

  // macro adds a nested type Set that is the option set
}

Doug

[tera]

@Douglas_Gregor

So, I went ahead and implemented all three.

That's really cool. Personally form 2 feels a bit heavy due to nesting, but I like all three. Repeating "UInt8" twice in option 3 is tolerable IMHO.

Some questions (sorry can't test it myself here):

• in form-1, can I use static var nextDay: Self specifying Self instead of explicit type?
• in form-1 could it be "let" instead of var? if "var" would I be able reassigning it?!
• in all forms, is it possible to customise the value, say set priority to be bit 5 instead of 2. Will standard get bit 6 automatically?
• I guess it won't be possible to customise the values to be a bit mask instead of a bit position (but that's probably minor and exceptionally rarely needed in practice)
• in all forms: what will happen if I add 9th element? (with UInt8 as the underlying type).

Plus:

• in all forms: can we have "all" value supported automatically without explicitly defining it?

(on the last bullet point: a bit dodgy "all" implementation defined on OptionSet)

extension OptionSet {
    static var all: Self {
        var v: Self!
        memset(&v, 0xFF, MemoryLayout<Self>.size)
        return v
    }
}

[esummers (Eric)]

@Douglas_Gregor

I found it interesting how we've had at least different ways in which we could describe option sets.

Option 3

It looks like SwiftUI chose method #3 for Edge.Set because they needed a single edge in some contexts and a set of edges in other contexts, but they are not well integrated despite being nested.

#3 I think would be more interesting if the enum synthesized RawRepresentable with single flag OptionSets so you might be able to use them to so some set algebra on the Edge.Set. It could be interesting to mix these single flag option sets with .allCases.

The downside is this is very complex, might need an implicit conversion (often frowned upon), and I'm not sure if a macro could synthesize all of this.

// based on #3
@OptionSet<UInt8>
enum ShippingOption: UInt8, ExpressibleByOptionSetLiteral { 
  case nextDay   // macro sets raw value to:  1 << 0
  case secondDay // macro sets raw value to:  1 << 1
  case priority  // macro sets raw value to:  1 << 2
  case standard  // macro sets raw value to:  1 << 3

  // macro adds a nested type Set that is the option set
}

Option 2

#2 could be made to work like form of #3. I think this makes #2 better than #3 unless #3 can somehow tightly integrate the enum with the set.

enum ShippingOption: UInt8 {
  case nextDay
  case secondDay
  case priority
  case standard

  @OptionSet<UInt8>
   struct Set {
      typealias Options = ShippingOption
   }
}

I think I prefer #2 or #3, but it is mainly because I have some use cases in a manifold mesh library I'm writing where I would need both an enum and a set and I'd rather not repeat myself.

[ksluder (Kyle Sluder)]

Option 3 seems to tacitly admit that set algebra is separable from the more primitive “flags” concept, which would be very useful on its own. I can envision a @Flags macro that optionally supported conformance to OptionSet:

struct MyView {
  @Flags
  struct ViewFlags {
    // Set algebra doesn’t make sense for these flags.
    var inWindow: Bool
    var hasSuperview: Bool
    var needsLayout: Bool
  }
  var flags: ViewFlags

  @Flags
  enum Edge : OptionSet {
    // Set algebra _does_ make sense for these flags.
    case leading
    case trailing
    case top
    case bottom
  }
}

It might even be the case that all @Flags enums are implicitly option sets, while all @Flags structs are not.

[esummers (Eric)]

@ksluder

Option 3 seems to tacitly admit that set algebra is separable from the more primitive “flags” concept

I think this too. It might be a good thing because you are less likely to abuse OptionSets if it gets enum baggage. It would still allow OptionSets to work the traditional way for interop.

I think it makes sense not to combine them in to one macro though. The "primitive flags" just needs memberwise initializers. If it uses a macro at all, it would just be for bit packing.

I'm very supportive of Swift adding memberwise initializers. Hopefully a way to do memberwise pattern matching and memberwise destructuring too. This has almost happened a number of times I think. Memberwise structs could allow a lot of cool stuff beyond just FlagSets:

let Point(x, y, z) = somePoint // destructures a point to x, y, z variables

[tera]

@Karl

Maybe that's a direction worth exploring for macros - so we could actually have something like:

optionset Flags: UInt8 {
  var a, b, c, d
}

And the macro would generate the appropriate struct, as a library feature.

I like that that very much!

I could see that kind of system having other uses, as well. For instance, SwiftUI views are not really like most structs because their properties usually have reference semantics via @State and @Binding, and the need to annotate all of those properties leads to views becoming very noisy. Perhaps they should be able to define a new kind of declaration via a macro:

view Map {
  var region: MKCoordinateRegion = ...
}

// would generate the same code as you write manually today:

struct Map: View {
  @State var region: MKCoordinateRegion = ...
}

// But it would eliminiate a lot of noise:
// - 'struct' becomes an implementation detail
// - conformance to 'View' is implicit
// - @State becomes the default

Yes! Perhaps not making @State default though, but:

view Map {
    state var region: MKCoordinateRegion = ...
}

where state is this new "meta-macro" getting translated into (or treated as) "@State"

With Doug's recent examples we could have a nice looking, as if it was a first class language construction on par with struct, enum, etc:

optionset ShippingOptions: UInt8 {

getting translated / treated as one of the following forms (TBD which):

// Form #1
@OptionSet<UInt8>
struct ShippingOptions {

// Form #2
@OptionSet<UInt8>
struct ShippingOptions {
  enum Options {

// Form #3
@OptionSet<UInt8>
enum ShippingOptions: UInt8 {

Could macro system be extended to support that? @Karl would you be interested making a pitch for that?

[Douglas_Gregor]

@tera

in form-1, can I use static var nextDay: Self specifying Self instead of explicit type?

Sure. It should work already.

@tera

in form-1 could it be "let" instead of var? if "var" would I be able reassigning it?!

It's going to turn into a computed property, so it can't be a let.

@tera

in all forms, is it possible to customise the value, say set priority to be bit 5 instead of 2. Will standard get bit 6 automatically?

For forms 2 and 3 where we have raw-valued enums, yes, you can customize the value and the compiler will pick the "next" value for subsequent cases.

@tera

I guess it won't be possible to customise the values to be a bit mask instead of a bit position (but that's probably minor and exceptionally rarely needed in practice)

We could certainly allow it. If we wanted customization for form #1, that's how it would have to be anyway.

@tera

n all forms: what will happen if I add 9th element? (with UInt8 as the underlying type).

Right now, it'll trap at runtime when you use that 9th element, but that's just me being lazy: so long as we know how many bits are in the underlying type (easy with sized integer types, problematic for Int/UInt), we could diagnose this in the macro. I intend to do this, but haven't gotten to it yet.

@tera

in all forms: can we have "all" value supported automatically without explicitly defining it?

That'd be easy to add.

@esummers

#3 I think would be more interesting if the enum synthesized RawRepresentable with single flag OptionSets so you might be able to use them to so some set algebra on the Edge.Set. It could be interesting to mix these single flag option sets with .allCases.

This seems doable, although the ExpressibleByOptionSetLiteral bit is stepping beyond "macro automating boilerplate for existing patterns" and toward "creating new ways to define option sets"; it might not be the best form for the latter.

@esummers

#2 could be made to work like form of #3. I think this makes #2 better than #3 unless #3 can somehow tightly integrate the enum with the set.

FWIW, putting the @OptionSet on the nested Set type won't work here, because the macro doesn't have visibility into the cases of the outer enum.

@ksluder

I can envision a @Flags macro that optionally supported conformance to OptionSet

I posted a similar @FlagSet notion earlier. I think it's a cool direction, and useful, and... not option sets.

Doug

[benrimmington]

@allevato

Option sets would have been an interesting API to explore, since that would have mapped nicely to the ICU flag sets for those scalars, but we would have been limited to 64 Boolean properties and IIRC ICU was already exceeding that at the time, or was close to it.

It seems possible to support larger option sets:

struct Options256: OptionSet {
  let rawValue: SIMD4<UInt64>
  static var lower: Self { Self(bitIndex: 0) }
  static var other: Self { Self(bitIndex: 123) }
  static var upper: Self { Self(bitIndex: 255) }
}

The init(bitIndex:) is custom API. Only four default implementations are needed:

extension OptionSet where RawValue: SIMD, RawValue.Scalar: FixedWidthInteger {

  public init(bitIndex: Int) {
    precondition(bitIndex >= 0)
    precondition(bitIndex < RawValue.scalarCount * RawValue.Scalar.bitWidth)
    let vectorIndex = bitIndex >> RawValue.Scalar.bitWidth.trailingZeroBitCount
    let scalarValue = RawValue.Scalar(1) &<< bitIndex
    var rawValue = RawValue()
    rawValue[vectorIndex] = scalarValue
    self.init(rawValue: rawValue)
  }

  public init() {
    self.init(rawValue: RawValue())
  }

  public mutating func formUnion(_ other: Self) {
    self = Self(rawValue: self.rawValue | other.rawValue)
  }

  public mutating func formIntersection(_ other: Self) {
    self = Self(rawValue: self.rawValue & other.rawValue)
  }

  public mutating func formSymmetricDifference(_ other: Self) {
    self = Self(rawValue: self.rawValue ^ other.rawValue)
  }
}

[esummers (Eric)]

@benrimmington

It seems possible to support larger option sets:

I think 64 options is already on the extreme side. Anything that wide is mainly doing it to align with pointer size. It is very rare for me to find OptionSets that need more than 8 options. The sets that get anywhere close are doing it for C-interop and those cases are generally restricted to 32 or 64 bit to match pointer size.

FlagSets are of course another story, but those should grow arbitrarily large. I could see multiple OptionSets packed in to a FlagSet. In fact, I’m pretty sure the FlagSet macro design that Douglas_Gregor presented earlier would allow clipping OptionSets down, For example, you could have a 7-bit option set and use the extra bit for a flag. You would just need to make sure you don't have 8 options or the last one will get chopped off!

[Douglas_Gregor]

@benrimmington

It seems possible to support larger option sets:

struct Options256: OptionSet {
  let rawValue: SIMD4<UInt64>
  static var lower: Self { Self(bitIndex: 0) }
  static var other: Self { Self(bitIndex: 123) }
  static var upper: Self { Self(bitIndex: 255) }
}

I think one could also use tuples of UInt64s to get a predictably-laid-out bag of bits that could be arbitrarily packed.

Doug

[aggie33]

The macro could avoid this scenario by checking the number of cases at compile-time.

public struct Options: MemberMacro {
    public static func expansion<Declaration, Context>(of node: AttributeSyntax, providingMembersOf declaration: Declaration, in context: Context) throws -> [DeclSyntax] where Declaration : DeclGroupSyntax, Context : MacroExpansionContext {
        guard let declaration = declaration.as(EnumDeclSyntax.self) else {
            fatalError("Cannot apply OptionSet macro to a non-enum type")
        }
    
        func makeAssociatedType() -> InheritedTypeSyntax {
            switch declaration.memberBlock.members.filter { $0.as(EnumCaseElementSyntax.self) != nil }.count {
            case 0...8:
                return InheritedTypeSyntax(typeName: TypeSyntax("UInt8"))
            case 9...16:
                return InheritedTypeSyntax(typeName: TypeSyntax("UInt16"))
            case 17...32:
                return InheritedTypeSyntax(typeName: TypeSyntax("UInt32"))
            case 33...64:
                return InheritedTypeSyntax(typeName: TypeSyntax("UInt64"))
            default:
                fatalError("Too many enum cases")
            }
            
        }
        let associatedType = declaration.inheritanceClause?.inheritedTypeCollection.first ?? makeAssociatedType()
        
        var index = 0
        var members: [String: String] = [:]
        
        for member in declaration.memberBlock.members {
            if let member = member.decl.as(EnumCaseDeclSyntax.self) {
                if let element = member.elements.first?.as(EnumCaseElementSyntax.self) {
                    if let value = element.rawValue?.value.as(IntegerLiteralExprSyntax.self) {
                        if let i = Int(value.description) {
                            index = i
                        }
                        
                        members[element.identifier.description] = value.description
                    } else {
                        members[element.identifier.description] = String(describing: index)
                    }
                    
                    index += 1
                }
            }
        }
        
        let string = members.map {
            "\tpublic static var \($0.key.trimmingCharacters(in: .whitespaces)) = Self(rawValue: 1 << \($0.value))"
        }.joined(separator: "\n\t")
        
        return ["""
        \n\n\tpublic struct Set: OptionSet {
            \tpublic var rawValue: \(associatedType)
            \tpublic init(rawValue: \(raw: associatedType.description.trimmingCharacters(in: .whitespaces))) {
               \t\tself.rawValue = rawValue
            \t}
            \t
            \(raw: string)
        \t}
        """]
    }
}

[benrimmington]

The @OptionSet macro appears to be implemented in the Swift 5.9 snapshots. Should it be removed until this proposal has been accepted?

https://github.com/apple/swift/blob/swift-5.9-DEVELOPMENT-SNAPSHOT-2023-05-26-a/stdlib/public/core/Macros.swift#L66-L94

[DevAndArtist (Adrian Meister)]

Why did this landed without a review?

OptionSet | Apple Developer Documentation

[Tino]

It is quite common nowadays that features are integrated without review, isn't it? This might be more prominent than some new capability that is hidden behind a compiler flag, but I think the official evolution documents just don't reflect reality (anymore).

Examples

@Joe_Groff

Noncopyable types can be experimented with using recent nightly toolchains and the compiler flag -enable-experimental-move-only, using the attribute name @_moveOnly for now. We hope you all will start to experiment with it!

@Ben_Cohen

The approach taken with the regex proposals (as with Swift Concurrency) is that the work is getting integrated under a compiler flag (-enable-bare-slash-regex in this case) while under review.

This topic seems to have become big enough for a separate thread...

[Ben_Cohen]

This landed inadvertently as part of the macros deployment and the plan is either to run a review or back it out (or keep it under an experimental flag – unlikely in this case due to its nature).

I don't think it's at all true that it's "quite common" to land changes without a review. Some minor changes are considered bug fixes, and some changes land with underscores or flags to ensure they can develop in-tree (especially if they are more complex in terms of their integration with the compiler).

The exception here is C++ interoperability which is fairly different in nature and so is taking a path outlined in John's post here.