Use SwiftSyntax itself to generate SwiftSyntax’s source code

Development Source Tooling
,
1

Hi @ahoppen.

I'm Vinícius Julião, from Federal University of Minas Gerais. I'm really interested in working on the SwiftSyntax project to replace gyb templates by the use of SwiftSyntax, more specifically SwiftSyntaxBuilder. While I've built SwiftSyntax I've found some issues in Swift, Foundation and even SwiftSyntax; for those SwiftSyntax issues I opened pull requests. See below

So far I don't have a proposed solution for problems related to Foundation and Swift core. So I'm going to post issue information on swift blog, in order to raise discussion that can help to solve those things.

Lastly, I've tried to implement a short use case for that project, where I'm using SwiftSyntaxBuilder to generate Tokens.swift, like it is done by Tokens.swift.gyb. See:

However, some initialization parameters of the class "ExtensionDecl" are not so clear for me. For example: What should extendendType be? I mean, in a swift code, what exactly is this extedendType? Could you help me with this @ahoppen? I think this example will help me to be familiarized with swift.

Furthermore, I'd like to ask you if there is something else I could read or practice to be familiarized with SwiftSyntax. Is it there?

A short about me: I’m about to graduate with a bachelor’s of Computer Science, with an expected graduation date for the end of this present year. I will start a master’s in Compilers next year, and I’d like to use the GSOC project in my portfolio for that goal. I’ve been involved in compiler research since 2020, where I could contribute to the creation of a new DSL for access policy purposes and successfully submit a paper. As an Open Source enthusiast, I’ve created and maintained a Visual Studio Code extension that integrates a StandardML repl with the text editor.

All the best,
Vinicius Julião R.

4 Likes
2

Hi @vrjuliao,

I’m glad to hear you’re interested in the SwiftSyntax GSoC project.

extendedType is the typename that follows the extension keyword. For example, if you want to generate

extension Foo {}

extendedType would be Foo. In general, if you can’t figure out what one of the parameters do, my advice would be to put something in that compiles, generate the code and see what it produces. Once you’ve figured it out, you could also add documentation for those parameters. For example https://github.com/apple/swift/blob/7553a37c285cb6d0ec81eb905bf665dd98810e6e/utils/gyb_syntax_support/DeclNodes.py#L283 doesn’t have a description, but https://github.com/apple/swift/blob/7553a37c285cb6d0ec81eb905bf665dd98810e6e/utils/gyb_syntax_support/DeclNodes.py#L311-L312 does.

I think you are on a good track. Just playing around with SwiftSyntax and trying to get something to work is the best way to familiarize yourself with it in my experience.

2 Likes
3

Hi @ahoppen!
Thank you for the tips, I finally could generate the Tokens.swift file using SwiftSyntaxBuilder.
Here is the generated file:

The code used for that generation are the two following ones:

However I toke some points regarding SwiftSyntaxBuilder while I was developing it, and I think such points could be part of the GSOC project scope. The points are:

  • There is no way to generate files like the gyb ones. I mean, I could not find any way to format a file, and also comment it. So, maybe, we could provide ways to do that. I can see that there is an auto-indentation tool, but we cannot generate a file like SwiftSyntaxBuilder/gyb_generated/Tokens.swift since it splits the chain of member calls SyntaxFactory.makeSpacedBinaryOperator(text).withLeadingTrivia(.spaces(1)).withTrailingTrivia(.spaces(1)) in multiple lines. I propose to add a TrailingNewLine attribute in some calls and declaraions in order to improve that formatting. For example: in a Chain of member calls, in an ArgumentList, or ArrayDecl for example.
  • Furthermore, we don't have an way to provide comments (neither blocks or inline).

I think, those are minor points, only formatting aspects, that would increase the power of SwiftSyntax. However, the last point I think it would be nice to discuss:

  • To create an expression like:
    static var `enum`: TokenSyntax{ ... }
    We have to nest: VariableDecl -> PatternBinding.accessor -> CodeBlock. Actually, I'm not sure if the expression inside the braces is an accessor (in the Tokens.swift file). It seems like an initialization. So, perhaps we could improve the InitializationClause in order to comply with it.

I'm sure that while the gyb file are migrated to SwiftSyntaxBuilder, more insights will occur, and SwiftSyntax become more powerful.

I'm preparing my proposal to send it in a couple of days.
The links you gave were really helpful. Thank you so much @ahoppen.
Sincerely,
Vinícius Julião R.

1 Like
4

I wouldn’t worry about formatting too much at this point. Formatting is a non-trivial problem to solve and I would suggest running a formatter like swift-format on the code generated by SwiftSyntaxBuilder but that would be a follow-up step after generating the actual code.

Yes, that’s a known limitation at the moment. @iMostfa has some ideas how to fix this in [GSoC 2022] Use SwiftSyntax itself to generate SwiftSyntax’s source code instead of GYB - #9 by iMostfa

The code is an accessor and not an initializer because every time you access the enum variable, the code in the braces gets executed to generate TokenSyntax representing the enum keyword. You can see this behavior if you e.g. put a print statement in the accessor. An initializer would be static var `enum`: TokenSyntax = SyntaxFactory.makeEnumKeyword().withTrailingTrivia(.spaces(1))

I agree that the syntax to generate variables is a little verbose at the moment. IMHO adding convenience initializers like the one you added in Tokens.gen.swift is the way forward here.

1 Like