[Pitch] Swift Predicates

There is a typo in one of the examples. An extra quote is in the middle of this string.

NSPredicate(format: "SUBQUERY(recipients, $recipient, recipient.firstName == sender.firstName").@count > 0")

Hi all, thank you everyone for your feedback so far! We've made some small updates to the pitch, and I've updated the pitch document linked in this post (the document here for reference). The main changes include:

  • Adding the full definitions of each expression operator
  • Renaming #predicate to #Predicate to align with capitalized names typically used for invoking a type's initializer
  • Substituting a new build_KeyPath function invoked by the macro for previous uses of dynamic member lookup

We appreciate all of your input, and we'd love any feedback you may have with these new revisions. Feel free to let us know if you have any comments or questions!

1 Like

I don't think we've established macro naming as part of the guidelines, but this seems to be an odd choice to me. If it starts with # it's a macro, whether or not that macro initializes a type under the hood. I'd expect macros to be lower cased. If you want to initialize a type, can't you make a normal Predicate type that takes the macro closure in one of the initializers? Frankly I'd prefer that rather than seeing # hanging around so much.

9 Likes

This is very interesting; having an example of multiple cutting-edge language features (variadic generics, macros) working in unison to tighten type-safety is compelling.

I'm curious how the set of operations supported by the Predicate macro system will be understood by developers. I see a couple avenues where this may arise:

  1. Diagnostics, when a developer attempts to use an operation not supported by the macro transform;
  2. Autocomplete support when typing a macro, as stated in goal (2) of the proposal's Motivation section.

For example, suppose a developer is interested in achieving the semantics of this example from the proposal:

let predicate = #Predicate<Message> { message in
    message.recipients.filter {
        $0.firstName == message.sender.firstName
    }.count > 0
}

but, seeing what looks like freely-written Swift in the provided closure, instead attempts to write:

let predicate = #Predicate<Message> { message in
    message.recipients.map(\.firstName).filter {
        $0 == message.sender.firstName
    }.count > 0
}

where map is an example of an unsupported operator (and presumably this fails to compile).

Can diagnostics & autocomplete inform the developer that e.g. the only supported operations on sequences are filter(_:) , contains(_:), contains(where:), allSatisfy(_:), etc.?

If supported in autocomplete, how is that relationship between macros and the IDE communicated? I'm guessing the macro definition itself isn't enough, but if that relationship is described in one of the macro proposals and I've missed it, my apologies.

(I recognize some developer experience-type questions may be outside the scope of this pitch, but thought I'd ask!)

1 Like

You're right, we haven't quite established naming as part of the guidelines (tagging @Douglas_Gregor since we've briefly discussed this). I don't think dropping the # altogether is a direction we want to go towards. While the semantics of the pre- and post-expansion code are the same, there's quite a bit of heavy lifting going on in the macro here that we'd like to be clearly evident to the developer by writing #Predicate at the construction site rather than hiding the macro invocation in the declaration of the initializer. Given the choice between something like #predicate and #Predicate, we felt that #Predicate looked more natural to indicate that the macro initializes a type, rather than something like #assert which acts more like a function call. Doug might have some more thoughts here.

2 Likes

Potentially a bit out of scope for the pitch here, but still an important question nonetheless! Currently diagnostics are our main tool here, and in fact this is one of the compelling reasons why we'd like to use macros instead of a solution like operator overloading. The macro will produce diagnostics for this case that tell the developer that the function is not able to be used in the context of the predicate they are creating. For common functions, we also have the opportunity to provide fix-its or suggestions as applicable within these diagnostics, but in general the diagnostic will just alert the developer that this function can't be used. Currently, our main source of truth for developers to see a list of supported operations would be the documentation (for example, predicates support all expressions that conform to StandardPredicateExpression). Macros don't currently have a way to influence the autocomplete results, but that is an avenue that could be interesting to bring up on the macro proposal.

2 Likes

This is a much larger discussion that doesn't really need to be here, so I've threaded it.

Macro Naming One simple objection is, why is a macro that returns a type (or otherwise looks like an init) different than a global or other function that does the same thing? If we had a global `predicate` factory, by this logic shouldn't it be `Predicate` (barring the inevitable collision with the real type)? Also, given that result builder closures are unmarked (despite feedback in the review, IIRC), why wouldn't we allow or expect macro closures to also be unmarked? If we're really that's concerned, adding a `#{}` form for macros (where the actual `#predicate` type is inferred) seems logical so we can have `Predicate {}` and `Predicate #{}` rather than `Predicate {}` and `#Predicate {}`. But then I may be in the minority that really doesn't want yet another marker / syntax for macros.
4 Likes

This is an interesting pitch which provides a lot of useful functionality!

So I have one fundamental question - what would be the approach for allowing for a UI that edits a representation of a predicate and transforms it back to the serialised entity that can be sent over to another process?

Specifically, we'd like to use predicates as a generic filter mechanism, where e.g. a frontend can edit a predicate (including values that are used as part of the expressions) and get a serialised representation that can be sent over the network to another process that then applies it (either directly or using the described support for transform to e.g. SQL).

It seems the needed mechanisms are halfway there ("Tree walking" section), but it is not (yet) quite clear to me if this would be flexible enough of a hook to go from serialised predicate -> UI. The reverse (going from some flexible/dynamic internal representation that allows a human being to interact with it, rather than coding stuff, over to the Predicate format) seems out of scope currently (but would be a critical piece to make this deeply more useful for many use cases).

Perhaps I'm missing something, but would be interested to hear your ideas here.

3 Likes

What do you see as the obstacles to doing what you propose? I think it's an entirely reasonable goal we want to support.

Not sure there are obstacles, I'm just trying to understand how do to a few things based of the pitch description (it's harder when you can't play with it :slight_smile: ) - maybe two questions to begin with:

  1. To confirm: to programatically (dynamically at runtime) generate a custom predicate tree, we'd use the low-level building blocks as outlined in the section "Macro processing" (that shows how the macros are expanded) ?
  2. In the 'tree walking' section, it is outlined how to add custom predicate processing - but how do you actually trigger a tree walk of myPredicate - is it by simply calling evaluate ? In that case, will a full evaluation always be done of all nodes in the tree, or can logic be short circuited before all nodes are traversed in the tree?
2 Likes
  1. Yes, you'd construct the tree using the public initializers on the various PredicateExpression types.
  2. As the tree walking section shows, this is typically done by defining a protocol, and then conforming Predicate and all the PredicateExpression types you want to handle to that protocol via extensions. If the protocol is (e.g.) ParseToResult like so:
protocol ParseToResult {
    func parse() -> Result
}

after writing extensions to conform Predicate et. al. to ParseToResult, you'd then do this:

let myResult = aPredicate.parse()

Note that the parse function could have an argument that would act as global state that could be passed down to component PredicateExpression values. While the example works if you have your own Predicate type, to do it for standard Predicate you need to cast:

extension Predicate: ParseToResult
    func parse() -> Result {
        return (expression as! ParseToResult).parse()
    }
}

If you wanted to allow for the case where you don't handle every PredicateExpression type, you could make parse() return an optional and use as? instead.

evaluate() is not involved in tree walking. You only use that if you wish to supply input to the predicate and evaluate it.

3 Likes

Thanks for the clarification @dgoldsmith (we missed parse() somehow, it was the missing piece for us - EDIT: in fact, looking at the pitch, I can't find it, perhaps it can be added to the Tree Walking section as an example? It would be clearer than spotlightQuery at least for us) - we've discussed the pitch with our team internally and overall we think it would be a great addition and definitely would make heavy use of this, looks really promising.

Our only remaining major concern (which obviously is hard to know from a pitch, but please view this as an open question) would as mentioned be performance (of evaluate() - specifically the use of keypaths, as even though SE-061 have the following note on performance:

The performance of interacting with a property/subscript via KeyPaths should be close to the cost of calling the property directly.

There are a few references so far to (quite significant) performance issues with the current keypath implementation, e.g.:

and

I understand the optimization of keypath handling is handled by a different set of priorities, I just wanted to point it out if there is anything that can be done to minimize that possible impact.

I guess it's only tangential to the pitch, but wanted to at least call it out as performance of evaluate() is critical to the usability of Predicates (at least for us) - and the pitch overall really looks great and we'd be super happy to use it (as long as performance is ok).

So, anyway - big +1 for the pitch overall.

3 Likes

Thanks for the explanation! I played around with expression macros and (although I got quite a few errors and couldn’t run anything) I understand why they’re used in the pitch. My only concern of macros in general, which does extend to this pitch, is the risk duplicating common functionality in different projects. For example, even after the advent of macros, property wrappers are still a great way of adding behavior to properties in a way most Swift programmers understand. I think this is the case with the proposal’s macros aa they are mainly used for custom operators. However, this behavior is not unique to predicates, the power assert discussed in the expression-macro threads is another great example and I imagine scoped operators being used for numerical computing too. In other words, you made a great case for why predicates would benefit from scoped operators instead of overloading, but we should generalize this feature to extend beyond Foundation. Otherwise the Swift ecosystem will become fragmented with each library author choosing their own version of scoped operators. The following is a simple, generic design we could use for this feature:

macro Predicate<R>(body: () -> R) = #scopedOperators(
  OperatorDescriptor(
    infix: “+”, 
    implementation: PredicateExpressions.Equal.init
  ),
  body
)
2 Likes

Quite an interesting idea. A similar idea, but for result builders, was recently pitched here:

Maybe a more general feature could solve both problems. I guess namespace and automatic usages of namespaces could be a nice thing but of course a lot of work to design and implement. I imagine this would also make autocomplete work more seamless.

2 Likes

I actually hadn’t considered a unified result-builder and operator namespacing feature; it’s a great idea!

The design would definitely be time consuming. For one, there’s the question of whether operators outside the namespace are just prioritized over global ones, or completely prohibited (e.g. there’s no bitshift operator in SwiftPredicates). However, the implementation, at least for the prototype in my previous post, was actually quite simple. You just parse the operators given to the macro, visit all operator nodes and substitute with the correct implementation.

2 Likes

Hi all, as mentioned in this pitch, we've posted a followup pitch regarding the serialization behavior of predicates at [Pitch] Swift Predicates: Archiving.

3 Likes

I'm glad to see this proposal went into live with Xcode 15 Beta, but I noticed that the variadic generic APIs are not implemented. In fact the variadic generic semantics are not fully described in the proposal, but it did mention an example of the macro and listed an entry in detailed designs.

So my question is: Is the variadic generic version of Predicate and #Predicate still planned? How soon can we use it on Darwin & how soon can we see it in swift-foundation?

1 Like

Yes! We still plan to adopt variadic generics / parameter packs for the new Predicate type. I don't have exact details that I can provide on when it'll appear in Darwin but I've posted a new and updated PR for swift-foundation at Variadic Generics Support for Predicate by jmschonfeld · Pull Request #178 · apple/swift-foundation · GitHub. I'm just working out a few final details before hopefully landing this.

3 Likes

Glad to see the PR being merged! @jmschonfeld

I've got another question: Why there's no Equatable and Hashable conformance to StandardPredicateExpressions? This is really useful when forming a fact list, etc.

If there's no particular reason for not adding the conformance, I wish to see it before the new OS fall releases, or else we may encounter the ABI hell.

2 Likes

Great question! I've thought about this a little bit when I was working on the initial implementation of Predicate. I chose not to add this for a few reasons. First, each conformance requirement on the expressions contained within a Predicate also translate to a conformance requirement on every value captured by a Predicate. We could require that every captured value be Hashable (in addition to the current Codable & Sendable requirement), but that's a choice we'd have to make if Predicate were also Hashable/Equatable.

However, likely more importantly, I didn't feel that the Equatable conformance for Predicate would actually be very useful in practice. Delegating down to an expression's Equatable conformance would check whether the structure of the expression is identical to that of another structure, but not truly equal. For example, consider the following two predicates:

// Predicate A
#Predicate<Bool, Bool, Bool> { ($0 && $1) && $2 }

// Predicate B
#Predicate<Bool, Bool, Bool> { $0 && ($1 && $2) }

These two predicates are semantically equivalent given they will always produce the same results and are equivalent boolean-algebra-wise. However, an Equatable conformance on the expressions would return that these are not equal because the expressions are actually entirely different types. Since Equatable doesn't afford for a way to "normalize" the expression tree before checking equality, in practice I suspect that an Equatable or Hashable conformance wouldn't produce desired results and likely wasn't worth the additional "cost" of requiring all captured values to be Hashable/Equatable as well.

5 Likes