Pitch: Static Custom Attributes (Round #2)

[anandabits (Matthew Johnson)]

@hartbit

I think this is an awesome idea! I'll work on it with the other co-authors.

Sounds good. I’d be happy to help flesh it out further as well. Pleas let me know if there is anything more I can do to contribute to this proposal.

@hartbit

How about dropping the scopes configuration for now? We can always add it back afterwards.

I think this would be a big mistake. I was very pleased to see this aspect of the proposal.

@hartbit

This makes me think that we should perhaps remove certain declarations like those until we are sure we can make them accessible through the tooling.

I strongly disagree. First, this is a language feature and should be defined on its own terms. There will be many uses for compile-time attributes. But secondarily, as an author of Sourcery templates I want to be able to be as precise as possible about valid use of my attributes. If tooling passes it through with some loss of context that is ok - I can still rely on the compiler having verified it.

If there are some declarations for which the tool does not make attributes available at all I would simply not use them until the tool is updated. I may file requests or even help contribute to the tool to make them available as I need them. If the language does not support these attributes this path is prematurely foreclosed: I would need to first return to SE and find an implementer interested in expanding the declarations supported by compile-time attributes.

If there is not a significant implementation cost to support granularity up front we should do it. Ease of use for those not interested in granularity can be supported using OptionSet. Limitations in granularity should be justified with a clear rationale (such as requiring significant incremental implementation effort, etc).

[akyrtzi (Argyrios Kyrtzidis)]

Just a suggestion, how about we start with a very restricted set of grammar rules and avoid full expression generality ? We can always consider as future direction to open the grammar more, e.g. perhaps once we have a builtin compiler-time interpreter that can be accessed by tools. We are not eliminating future directions, it is always easier to remove restrictions than start unrestricted and try to add restrictions later.

For example, say that in the initialization grammar we accept only a simple literal or enumerator reference expression, in the argument position, plus variadic arguments. This would allow tools to do simple syntactic interpretation. Then with this restrictions in mind you could check how many of the Sourcery templates it can cover.

[beccadax (Becca Royal-Gordon)]

I think it's a mistake to imagine the argument expressions as something that will be executed or evaluated in any sophisticated way at compile time. Remember that one of the important use cases we're imagining is annotating code for source synthesis tools. That means that, when you're parsing the code to interpret the attribute, the code isn't complete and might not even compile. Since we imagine attributes referring to enums and other declarations from the surrounding context, we can't evaluate these expression isolated from their context, and since their context might not be complete, we can't really evaluate them in their context either.

At the same time, I don't think it makes sense to severely restrict the kinds of expressions to, for instance, literals and references to enum cases. Giving tools flexibility is a good thing. In some cases, they may even want arbitrary expressions to drop into source code. Most tools will look for one particular kind of literal expression or leading-dot setup, but if they want to do something else, they can. The fact that we can give them a syntax tree and let them interpret it with as much or as little sophistication as they care to implement is a good thing.

[anandabits (Matthew Johnson)]

Even if we can’t evaluate attribute initializers right now what makes this more difficult than other forms of compile-time evaluation? It seems like this should work fine as long as everything is @compilerEvaluable and there are no circular dependencies involved.

[beccadax (Becca Royal-Gordon)]

There are fundamental parts of the compiler's structure that make it difficult.

Think of the compiler as a pipeline. For this discussion, I'll simplify that pipeline to the following:

Parsing → Analysis → Optimization → Machine Code Generation

First we parse the code to form a syntax tree and an AST; this is the level that SwiftSyntax works at. Then we analyze the code (in two separate parts, called "semantic analysis" and "mandatory SIL passes", but the distinction isn't important here) to check and infer types, synthesize implementations, bind names to declarations, and generally dot all the T's and cross all the I's in it; by the end of analysis, we've diagnosed basically all errors. Then we optimize the code (again, in two separate parts, but that's irrelevant). Finally, we actually produce machine code and other outputs.

@compilerEvaluable is part of optimization. It finds parts of the code that it knows how to evaluate at compile time and does so. But optimization doesn't happen until after analysis, and if an error is discovered during analysis, optimization never happens at all. One of the optimizer's invariants is that it is only run on well-formed code; a lot of crashes and weird misbehaviors you get from the compiler are the result of analysis failing to notice that code is invalid or mis-understanding the code and annotating it with information that makes it invalid, allowing that invalid code to be fed into later parts of the pipeline that aren't ready for them.

That means that, if you can only understand attribute arguments by running them through @compilerEvaluable, you can only understand attribute arguments in valid programs. Which sets up a nasty circularity: if a program is not valid without code generated by a tool, you can't generate the code because the program isn't valid, and you can't make the program valid without generating the code.

You might suggest the the solution is to only analyze and execute the parts of the program that the attribute references, but this is quite difficult to do. How to correctly compute the dependencies of a given piece of Swift code is, in fact, an open research question, and one that won't be answered terribly quickly.

Finally, it's worth keeping in mind that even if these issues can eventually be overcome, providing the evaluated value in addition to the parse tree is an additive change, and not even one that has to go through Evolution. It's just a little side information we can choose to add at any time.

[Vinicius_Vendramini]

From all that’s been said, I think limiting arguments to literals and enum references (with only literals also as associates values) would probably be enough for now. I believe current use cases can all be represented in this way, and that any future uses that require more complicated logic can be considered later.

Could anyone familiar with sourcery and swiftlint comment on whether or not these tools would need something else? From my uses of swiftlint this sure seems to be enough.

Also, if anyone is familiar with the interpretation that already exists in the compiler’s internal attributes, it’d be interesting to consider if that could be merged or extended into whatever we propose here.

[anandabits (Matthew Johnson)]

Thank you for such a detailed explanation @beccadax, this makes sense. It would be exciting to be able to just decode the attribute in a Sourcery template to get a value of the attribute type, but it sounds like that needs to remain a future direction (that may or may not ever happen), at least if we support arbitrary expressions in the attribute initializer.

@Vinicius_Vendramini

From all that’s been said, I think limiting arguments to literals and enum references (with only literals also as associates values) would probably be enough for now. I believe current use cases can all be represented in this way, and that any future uses that require more complicated logic can be considered later.

Could anyone familiar with sourcery and swiftlint comment on whether or not these tools would need something else? From my uses of swiftlint this sure seems to be enough.

I would expect Sourcery to just pass through attribute data to the template it is running. Custom annotations currently only support string, numeric and boolean values. I think most template authors expect to be able to get the value associated with the annotation easily without custom parsing logic. The current approach is to perform a dynamic cast on the values in the [String: NSObject] annotation dictionary.

If we want to support these templates well and convince Sourcery users to adopt custom attributes as a replacement for comment annotations we need to offer a similarly straightforward approach. If the compiler isn’t able to interpret expressions in attribute initializers I think a restriction is warranted, at least as an opt-in choice for the attribute author. We don’t want Sourcery or Sourcery templates attempting to do that.

If we introduce (possibly opt-in) restrictions on the expressions that may be used in custom attribute initializers that might make it easier to serialize attribute data. Instead of using Codable there could be a standard way of converting initializer argument labels and literal argument values to data. We could maybe even go so far as to provide a compiler-synthesized init(attributeData: Data) for compile-time attributes that opt-in to this restriction. This would allow the attribute value to be serialized by the compiler and deserialized by tools without requiring sophisticated compiler evaluation or fragile parsing logic in the tools.

This approach would let us start simple while still serializing a representation of the attribute. Over time the limitations on the kinds of expressions allowed when initializing these attributes could be relaxed. Eventually we might be able to support all @compilerEvaluable expressions.

[hartbit (David Hart)]

@Vinicius_Vendramini

From all that’s been said, I think limiting arguments to literals and enum references (with only literals also as associates values) would probably be enough for now. I believe current use cases can all be represented in this way, and that any future uses that require more complicated logic can be considered later.

I agree.

@anandabits

I would expect Sourcery to just pass through attribute data to the template it is running. Custom annotations currently only support string, numeric and boolean values. I think most template authors expect to be able to get the value associated with the annotation easily without custom parsing logic. The current approach is to perform a dynamic cast on the values in the [String: NSObject] annotation dictionary.

If we limit to literals, SourceKit should be able to provide the values directly in its JSON output.

One problem though: if we only support literals and enum cases without associated values, how do we define the @compileTimeAttribtue with an OptionSet parameter in its initializer? @Douglas_Gregor Is an OptionSet defined with a literal array parseable?

[anandabits (Matthew Johnson)]

@hartbit

If we limit to literals, SourceKit should be able to provide the values directly in its JSON output.

Even if SourceKit provides the values directly in the JSON output I still think it would be useful to have a compiler-synthesized init(attributeData:) or init(attributeJSON:). The compiler defines the encoding of the attribute data. It would be more convenient for users and also avoid potential for subtly incorrect decoding if the compiler also defined the decoding of the attribute data.

This approach would also be forward compatible with loosening the restrictions on supported initializer expressions. We should make design decisions with the future in mind. I really don’t think the right long-term decision is just exposing a JSON dump that tools with knowledge of specific attributes (such as a specific Sourcery template) have to parse manually.

@hartbit

One problem though: if we only support literals and enum cases without associated values,

Is there any reason you can’t support enums with associated values so long as the associated values themselves are literals or enum cases?

[quote="hartbit, post:48, topic:22938"]
how do we define the @compileTimeAttribtue with an OptionSet parameter in its initializer? @Douglas_Gregor Is an OptionSet defined with a literal array parseable?

I think it’s ok if @compileTimeAttribute cannot itself be defined as a custom @compileTimeAttribute with the current limitations. The recursive definition is kind of cool but should not be a driver of the design IMO. This attribute will be implemented using compiler magic anyway.

[marcelofabri]

I agree that the parameters should be limited to a subset that tools can easily parse as suggested (only literals, enums without associated values)

However, I'm not sure how useful this would be for SwiftLint, at least for suppressing warnings. I imagine people would still want the ability to enable/disable rules in regions of code where you can't add an attribute (e.g. a single try! inside a function).

Interestingly, Android Lint (one example of linter in a language with a similar concept than attributes) doesn't use enums to represent its rules: @SuppressLint("NewApi") (Improve your code with lint checks  |  Android Developers). Not sure if this was a limitation that they've faced or a conscious choice to keep the attributes simple.

On the other hand, I can imagine attributes being used for other purposes. We could for example have a @Pure attribute that a tool (i.e. SwiftLint) parse and validate that a function is pure (no side effects happen). I'm sure there're other similar use cases.

I'm also excited for what this means to Sourcery. We use several comment/annotations that would be better expressed with compiler support.

PS: I'm one of the maintainers of SwiftLint, but this is my personal opinion, not necessarily reflects the project direction, etc.

[Douglas_Gregor]

@anandabits

I still think the positioning for this case is not right. Swift went away from having keywords for parameters/attributes. Everything goes onto the parameter type. I don't think we should re-introduce this old behavior with this proposal and require consistency.

The attributes you refer to were moved because they really apply to the type of the parameter, not the parameter itself. It is certainly possible somebody has a use case for applying a custom attribute directly to a parameter. I don’t see why it should be banned just because none of the built-in attributes make sense in this position.

There are definitely use cases out there. You could express invariants on a parameter that aren't describe in the type system, like "this mutex is in the locked state."

I've flip-flopped on this, but I think we should support parameters, and make it clear that these attributes are for the parameter declaration---they are not part of the full function type.

Doug

[akyrtzi (Argyrios Kyrtzidis)]

@hartbit

One problem though: if we only support literals and enum cases without associated values, how do we define the @compileTimeAttribtue with an OptionSet parameter in its initializer?

For the case of passing a variable number of arguments in the attribute, I'd recommend considering variadic arguments, which would provide a straightforward syntactic tree for tools to interpret.

[johnfairh (John Fairhurst)]

@hartbit

SourceKit should be able to provide the values directly

I don't think we need to worry about changing this: the reason the current design works is that the client has the source file loaded anyway in order to do stuff with it and so can easily look up the value from the provided range.

[Alejandro_Martinez]

I like the way this looks

I'm wondering if the declarations and scopes used in the attribute (.struct, .class) can be accessed outside that attribute declaration? They seem to be defined in the stdlib so should be possible I assume. Even if they are not useful on their own it may be interesting to be able to explore them as a developer.

Thanks for clarifying the Namespacing rules. At a first glance I immediately thought that @Ignore should be @Linter.Ignore. Great job

Am I the only one that finds the static Scope name a little confusing?

Static custom attribtues (<- typo BTW) are powerful enough to allow several compiler-defined attributes to be re-defined inside the Standard Library.

When you say this, is it totally correct? I mean, yes, you can define them in the standard library, but what the attributes do still needs to be known by the compiler correct? I think that may be misleading when the proposal is official.

I'm really excited with this feature making the first steps to more compiler customisation. I know that some stuff is really far away but I hope we don't stop here. This can be cool to improve the current code gen, but I still thing that performance wise it will be wasteful to have 2 tools (code gen and compiler) do part of the same job twice. I would hope eventually we can use this custom attributed to hook into compiler to vend some code gen directly to it. But anyway, I digress, this is a great approach to custom attributes!

I love that they're just structs and that the only "special syntax" is the staticAttribute itself which seems fair because scopes and declarations is not something relevant outside of this.

[Vinicius_Vendramini]

@Alejandro_Martinez

Am I the only one that finds the static Scope name a little confusing?

I must admit that I also don't think the scope names are very clear here, but it might just be because we rarely have to think about them this way. That said, if you come up with any suggestions, I'd be happy to hear them.

@Alejandro_Martinez

I mean, yes, you can define them in the standard library, but what the attributes do still needs to be known by the compiler correct?

Yeah, the compiler's attributes change the behavior of the language (while the user attributes don't) and that behavior has to be implemented inside the compiler itself.

[Alejandro_Martinez]

@Vinicius_Vendramini

I must admit that I also don't think the scope names are very clear here, but it might just be because we rarely have to think about them this way. That said, if you come up with any suggestions, I'd be happy to hear them.

Reading them again I can see how is not that easy to pick a name for that case. I would say type scope vs. instance scope has a good similarity to "class methods" vs. "instance methods", which is the way I was taught.

[anandabits (Matthew Johnson)]

I thought of another issue that I think we need to consider in this discussion.

When a tool such as SwiftLint or a Sourcery template defines attributes it wants to process what is the recommended approach to making those attributes available in user code?

It doesn't seem to make much sense for users to have to link a framework that only includes static / compile-time attributes. Further, creating and distributing a framework so support something such as a small as a Sourcery template seems like overkill. The alternative is to have users paste the attribute declarations into their own source code. Some Sourcery templates require users to do this with protocol declarations today and would continue to rely on a supporting protocol regardless of how attributes are handled.

Are there any other options available? This seems to call for a kind of lightweight, compile-time-only dependency that doesn't exist yet. I can imagine this kind of dependency becoming something more broadly desirable as Swift increases its support for static / compile time metaprogramming. Any thoughts?

[rumnat (Muratov Anton)]

Would be great to continue the discussion on this topic

[Vinicius_Vendramini]

Hi @rumnat,

From what I remember from my conversations with @hartbit and @Douglas_Gregor, we were pretty comfortable with our initial design, but all of us were too busy at the time to get the implementation done so that we could turn this into a proper proposal. This was a while ago, however, and Swift has changed since then, so it might be worth revisiting the design.

I'm still very interested in this feature but, unfortunately, still too busy to implement it. If anyone wants to pick it up, I'm happy to help in any way I can.

[rumnat (Muratov Anton)]

I'd like to help but I've never done any proposal and not that familiar with the compiler and programming language theory. If someone is mentoring me I'd glad to help and take it :)