Update on the new Swift Parser

Hi all,

It's been a month since we introduced the new Swift parser into swift-syntax. Since then, we've made some great progress, so I wanted to share some of that here:

  • The new parser is now linked into the compiler, with additional testing that we can enable for every single compile. There are two kinds of tests at the moment:
    • Round-trip testing: this ensures that the syntax nodes produced by the new parser reproduce every byte of the input source correctly, no matter how broken the input source is. This critical testing for the robustness of the parser and syntax nodes.
    • Valid-parse testing: this ensures that, if the C++ parser produces no errors when parsing, the new Swift parser also produces a syntax tree that contains no unknown, unexpected, or missing nodes (all of which imply errors). This makes sure that the new parser is covering the full grammar correctly.
  • The compiler's full test suite (~15,000 tests) runs with both checked modes above enabled. There are only about two failures, which are intentionally-crafted bad inputs with, e.g., a couple of hundred opening ('s in a row.
  • Parser recovery is improved, including handling source files with malformed UTF-8.
  • Added support for creating syntax nodes via string interpolation, so one can create a syntax node by interpolating bits of syntax into a string literal, e.g.,
    let expr: ExprSyntax = 
      """
      \(node.withoutTrivia()).toggle()
      """
    
  • Added the SwiftOperators library, which handles operator precedence results to rewrite a syntax tree for a "sequence" expression such as x + y * z into one that shows the order of operations.
  • swift-format has adopted the new parser and SwiftOperators module.

Thanks to everyone who tried it out, provided feedback, and fixed bugs. Our next big leap forward is bringing up ASTGen, which will be part of the Swift compiler itself that translates Swift syntax nodes into the C++ AST, and will eventually replace the C++ parser.

Doug

69 Likes

Great news, indeed! It would be great to add lots of weird mistakes and check for good error messages. Or even typos that still parse but confuse the type checker...
Maybe telemetry to see how people correct errors?
(Thanks for letting me blue-sky your priorities.)

1 Like

Right. We’ve got a big pile of inputs from the C++ parser test suite to work through, and the goal is that it’s very easy to go add a new special-case diagnostic when we see a new weird case.

Doug

4 Likes

Really exciting updates and great work on the stability! But how about performance; has it been tested against the current parser?

Yes, we've been testing performance, and @rintaro in particular has made significant improvements here.

One recent data point: SwiftLint is seeing 0-8% speedup in optimized builds when replacing SwiftSyntaxParser with SwiftParser, per [WIP] Use SwiftSyntax's new SwiftParser by jpsim · Pull Request #4216 · realm/SwiftLint · GitHub.

Doug

4 Likes