October Update on the new Swift Parser

Hi everyone,

It’s now been a little more than two months since we introduced the new parser to swift-syntax. We have made some great progress since the last update:

  • We significantly improved the recovery and diagnostics of the parser:
    • 40 hand-crafted diagnostics were introduced to the parser
    • The majority of diagnostics for missing or unexpected tokens can be automatically generated with accomponying Fix-Its. For example parsing a standalone for i auto-genrates an diagnostic with text expected 'in', expression, and body in 'for' statement and a Fix-It that transforms the code to for i in <#expression#> {}.
    • We imported the old parser's test suite into the new parser, started off with 1821 TODOs for diagnostics that either needed to be ported to the new parser or deemed as semantic and thus later diagnosed in ASTGen. 1469 of these have been resolved, leaving 352 remaining.
  • We added an initial ASTGen library to the compiler, which will translate the parsed syntax trees into the C++ Abstract Syntax Tree (AST) used by the rest of the compiler.
  • Felix Wehnert added a pretty-printer of parser diagnostics to the swift-parser-cli testing tool that prints the diagnostics as ASCII art:
    1 │ foo.[].[].[]
    ∣     │  │  ╰─ expected name in member access
    ∣     │  ╰─ expected name in member access
    ∣     ╰─ expected name in member access
    
  • SwiftLint switched to the new parser rewrote more than 100 rules from SourceKit to SwiftSyntax.
  • Sourcery also migrated from the legacy C++ parser to the new parser.
57 Likes

I'd like to highlight a few of the benefits we've seen by moving SwiftLint to use the new Swift Parser:

  1. Up to ~15% faster lint times on macOS (not measured on Linux)
  2. Significantly smaller binaries: portable_swiftlint.zip is 11.3 MB down from 20.9 MB
  3. No need to vendor the static lib_InternalSwiftSyntaxParser library and the various linker hacks needed to support that
  4. Prevent Linux distribution errors like SwiftSyntax comment parsing error since 0.47.0+ on Linux builds · Issue #4031 · realm/SwiftLint · GitHub
  5. Clearly the future of SwiftSyntax considering how aggressively Apple is trying to remove the C++ parser: See Recore deprecated SwiftSyntaxParser on top of SwiftParser. by DougGregor · Pull Request #767 · apple/swift-syntax · GitHub
  6. Possibly a lower barrier for new contributors given that it's 100% Swift

These are some really nice wins, so thanks to the SwiftSyntax folks for making it happen.

The main downside that we only discovered after moving over to the new parser is that memory usage has increased by a factor of 2x to 4x compared to the 5.7 SwiftSyntax release. Because we've been migrating more and more SwiftLint rules from using SourceKit to use SwiftSyntax, this has amplified the impact and now the latest SwiftLint release candidate (0.50.0-rc.3) takes about twice the amount of memory than the last stable release (0.49.1). If you're processing tens of thousands of files, this can easily mean GBs of increased memory usage.

I'm not comfortable cutting a new stable SwiftLint release until we can find a way to reduce the memory usage back to a reasonable level. Any help here would be greatly appreciated. I'm sharing this so others considering using new versions of Swift Syntax are aware.

28 Likes

FYI, @rintaro has some ideas to address the issue that he'll be working on implementing.

Doug

5 Likes

That's great news, Doug. Thanks for the heads up.