A New Swift Parser for SwiftSyntax

[FranzBusch]

I haven't really looked into swift-package-factory but once SwiftSyntax adopts SemVer you should be able to just up the your required version of SwiftSyntax to one that supports PATs and you can generate your code. Since SwiftSyntax becomes decoupled from the toolchain there is no requirement on the user side except doing a package update to get the latest release of SPF.

[taylorswift]

the lower bound of the swift-syntax dependency would still become the lower bound of the consuming package’s toolchain support range. for example, right now swift-grammar compiles on 5.3, if swift-syntax requires 5.4 or higher, swift-grammar would have to drop support for 5.3.

[Douglas_Gregor]

@gwendal.roue

I'm talking about successfully building a package with swift-tools-version:5.8, and a dependency on SwiftSyntax for Swift 5.8, with a later compiler (Swift 5.8.1, Swift 5.9, and maybe more): this is a scenario where the version of SwiftSyntax lags behind the compiler version. Since this scenario goes further than what is strictly needed for a compiler, I feel the need to ask.

At some point, when we've audited all of the grammar productions, integrated the new parser into the compiler, and are generally happy with the SwiftSyntax API, we can take away this big warning:

Note: SwiftSyntax is still in development, and the API is not guaranteed to be stable. It's subject to change without warning.

and enable library evolution for SwiftSyntax. That should make it possible for a tool designed with (say) Swift 5.8 to still work with a Swift 5.9 toolchain.

@taylorswift

@retro only works if swift-syntax can understand PATs. and only recent snapshots of swift-syntax can understand PATs. so if you wanted SPF to run as a buildTool and not a command plugin, that would require the user to have a toolchain installed that understands PATs, which would defeat the purpose of @retro.

The new Swift parser's design should allow for some forward compatibility, because things that it doesn't recognize will be left in the tree as "unexpected" nodes. So, your

protocol ParsingRule<Element, Index> {
  // ...
}

example would still be parsed as a ProtocolDeclSyntax, with all of the tokens <, Element, ,, Index, and > showing up as unexpected nodes within that ProtocolDeclSyntax.

@igor-makarov

What are the implications of this on the question of adding macros to the compiler?

Having a new parser and syntax could make it easier to explore macros, although you would need a bunch more of the compiler (e.g., something that can load modules, reason about macros, and do name lookup) to do so. I hope that having the parser and syntax tree more readily available will make this kind of experimentation easier.

Doug

[FranzBusch]

I think we have been talking past each other a bit and the confusion comes from the shape of the code gen tools we are talking about. There are two kinds:

1. The ones that parse Swift code and generate more Swift code
2. The one that parse other file formats (e.g. proto files) and generate Swift code

For the second one, as long as you are only generating code that is using source stable features you should be totally fine as a build tool. The first one is more interesting because as you noted the version of SwiftSyntax you are using might not be able to understand the code. However as @Douglas_Gregor pointed out the library will output these as "unexpected" instead of just failing so you can handle this gracefully.

@taylorswift

the lower bound of the swift-syntax dependency would still become the lower bound of the consuming package’s toolchain support range. for example, right now swift-grammar compiles on 5.3, if swift-syntax requires 5.4 or higher, swift-grammar would have to drop support for 5.3.

But this is quite normal right? For example, most packages in the Swift on Server ecosystem depend on NIO. NIO always supports the latest two Swift versions. Once we drop support for a version that has effects on the whole ecosystem.
IMO SwiftSyntax needs to adopt a similar policy to also allow it to evolve its code base while still maintaining support for old Swift toolchains.
One more important point, if SwiftSyntax supports SemVer it should not create a new major for every new Swift version. This would effectively have the same effect has the current tagging mechanism.

[taylorswift]

@Douglas_Gregor

example would still be parsed as a ProtocolDeclSyntax, with all of the tokens <, Element, ,, Index, and > showing up as unexpected nodes within that ProtocolDeclSyntax.

SPF is very conservative about how it applies @retro, it will only delete syntax if it knows beforehand (from SwiftSyntax) that the node it is deleting is a valid PAT list.

SPF could be more aggressive and just delete every node the linked version of SwiftSyntax doesn’t understand. but to be honest i would not be very comfortable using a sourcegen tool that just deletes syntax it does not understand, and whose definition of "syntax it does not understand" changes depending on where the repository has been cloned.

@FranzBusch

But this is quite normal right? For example, most packages in the Swift on Server ecosystem depend on NIO. NIO always supports the latest two Swift versions. Once we drop support for a version that has effects on the whole ecosystem.

just because it is common doesn’t mean it’s a good idea. there are more people stuck using ancient toolchains than a lot of us in the server bubble assume.

if a package builds on 5.3, SwiftSyntax requiring 5.5 just isn’t a good enough reason to drop everyone using 5.3 if the package would still build if only we bothered to ship the generated .swift files with the repository.

[Douglas_Gregor]

@taylorswift

@Douglas_Gregor

example would still be parsed as a ProtocolDeclSyntax, with all of the tokens <, Element, ,, Index, and > showing up as unexpected nodes within that ProtocolDeclSyntax.

SPF is very conservative about how it applies @retro, it will only delete syntax if it knows beforehand (from SwiftSyntax) that the node it is deleting is a valid PAT list.

SPF could be more aggressive and just delete every node the linked version of SwiftSyntax doesn’t understand. but to be honest i would not be very comfortable using a sourcegen tool that just deletes syntax it does not understand, and whose definition of "syntax it does not understand" changes depending on where the repository has been cloned.

In my suggestion, SPF would have its own mini-parser to recognize the PAT syntax within the unknown nodes, which is very different from deleting every node that the linked SwiftSyntax doesn't understand.

Or SPF can vendor copy of the latest-and-greatest SwiftSyntax.

Doug

[MEnnabah (Ennabah)]

@Douglas_Gregor

Get involved!

The new parser is under active development, and we’d love help! You can experiment by pulling the latest SwiftSyntax and trying some code as shown in the quickstart guide . Testing is extremely important for such a central component. If you’d like to help out, you can run swift-parser-test on your own Swift code and report any bugs you find , optionally reducing them down to smaller test cases first or even taking a shot at fixing parser bugs directly. If you’re ready to dive into development, check out our implementation status page for ideas.

Not sure if it's the right thing to ask a question here or create a new topic. So if it's not, please let me know and I'll move it a new topic
I'd like to get involved, but I'm having some difficulties following FixingBugs.md.

From the docs:

Once you’ve written a test case (see below), set a breakpoint in Parser.parseSourceFile and navigate the debugger to the place where the parser behaves unexpectedly.

I used Ninja to build Swift (and --swiftsyntax --swiftpm --llbuild later to build SwiftSyntaxt). Then, I created .xcworkspace (in swift-project) and added a folder reference to swift-syntax.

The goal is to be able to set breakpoints, and navigate the debugger and execute lldb commands.

But I found 2 issues:

• AssertParse's parseSyntax argument emits error Cannot convert value of type 'RawSourceFileSyntax' to closure result type 'Node'. It suggests to cast it to Node.
  So eventually, the fixit is: $0.parseSourceFile() as! Node.
• Most of the AssertParse tests under SwiftParserTest couldn't be compiled, unless they pass the parseSyntax parameter. Otherwise, this error is emitted: Generic parameter 'Node' could not be inferred. (I've added a screenshot).

I'd like to contribute to SwiftSyntax, it would be great if anyone can hook me in! And please let me know if the workflow seems to be invalid.

[rintaro (Rintaro Ishizaki)]

Thank you for being interested in contributing to SwiftSyntax!

I confirmed the issue in Xcode 13.4, but not in Xcode 14 beta.
I think Unify all parser tests to use an enhanced version AssertParse by ahoppen · Pull Request #641 · apple/swift-syntax · GitHub introduced the issue (cc: @ahoppen) which we should fix.
Meanwhile, could you try using Xcode 14 beta?

[MEnnabah (Ennabah)]

Thanks for the help! I'll switch to the beta, and hopefully the 2 issues won't be there.

[MEnnabah (Ennabah)]

Yep it did solve both issues -- Many thanks!

[codafi (Robert Widmann)]

I am restoring the ability to build SwiftSyntax in Xcode 13 here Restore Xcode 13 Build by CodaFi · Pull Request #702 · apple/swift-syntax · GitHub

I apologize for the inconvenience. We're looking into expanding our continuous integration checks to account for situations like this.

[MEnnabah (Ennabah)]

Thanks Robert!

We're looking into expanding our continuous integration checks to account for situations like this.

On the Swift CI, I see there are build commands that trigger Xcode.app, and others trigger Xcode-beta.app. So I expected the CI builds for both, and if one fails it would report it (thus, it accounts for similar situations).

The snippet made me think it builds for Xcode 13 is:

/Users/ec2-user/jenkins/workspace/swift-syntax-PR-macOS/branch-main/build/buildbot_incremental/toolchain-macosx-x86_64/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swift build --package-path /Users/ec2-user/jenkins/workspace/swift-syntax-PR-macOS/branch-main/swift-syntax --configuration release --build-path /Users/ec2-user/jenkins/workspace/swift-syntax-PR-macOS/branch-main/build/buildbot_incremental/unified-swiftpm-build-macosx-x86_64 --multiroot-data-file /Users/ec2-user/jenkins/workspace/swift-syntax-PR-macOS/branch-main/swift/utils/build_swift/resources/SwiftPM-Unified-Build.xcworkspace --verbose --product SwiftSyntaxParser

Maybe I'm wrong in thinking Xcode.app is actually Xcode 13 tho. I might dig deeper to the CI to understand the current workflow.

[Douglas_Gregor]

Hey all,

Now that it's been a month, I posted an update on this project.

Doug