# Speeding up SwiftSyntax by using the parser directly **URL:** \ **Category:** Compiler\ **Created:** [December 1, 2018, 11:13pm UTC](https://forums.swift.org/t/speeding-up-swiftsyntax-by-using-the-parser-directly/18493 "2018-12-01T23:13:31Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![akyrtzi](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/akyrtzi/32/4963_2.png) [@akyrtzi](https://forums.swift.org/u/akyrtzi)\ **Post date:** [December 1, 2018, 11:13pm UTC](https://forums.swift.org/t/speeding-up-swiftsyntax-by-using-the-parser-directly/18493/1 "2018-12-01T23:13:31Z")
**tl;dr; Providing direct access to the parser speeds up SwiftSyntax 8x, and it becomes 2x faster than the legacy sourcekitd syntactic request** Currently there are 2 ways to create a SwiftSyntax tree from a source file: - Parse the JSON output from the compiler executable (slow) - Use sourcekitd + binary serialization format (fast) When comparing SwiftSyntax with the legacy request, a major benefit is that SwiftSyntax allows for incremental re-parsing but it is still critical for SwiftSyntax to have very fast, real-time performance when parsing a lot of code (e.g. first time a file is opened, processing multiple files, etc.). From that aspect, even the fastest available approach for SwiftSyntax is multiple times slower than the legacy sourcekitd syntactic request. To see how faster SwiftSyntax can be, I prototyped providing a C library that exposes a couple of APIs that use the Swift parser to create the raw syntax-tree nodes that SwiftSyntax needs, and pass them to the client. I then modified SwiftSyntax to call these C APIs and get back the raw syntax nodes directly from the parser. The performance gains of this approach are significant, see the timings below. For the source test case I used [the largest file in firefox-ios repo](https://github.com/mozilla-mobile/firefox-ios/blob/master/Client/Frontend/Browser/BrowserViewController.swift) and copied it 2x, so total is a file with 4,120 LOC. | | compiler side | SwiftSyntax side | total | | --- | --- | --- | --- | | legacy syntactic request | 25 ms | | **25 ms** | | SwiftSyntax/sourcekitd | 44 ms | 60 ms | **104 ms** | | SwiftSyntax/direct parse | 11 ms | 2 ms | **13 ms** | As you can see, using the C library SwiftSyntax became **8x** faster, and even **2x** faster than the legacy sourcekitd syntactic request! 馃弾馃挩 The only potential downside of this approach is that we are losing the crash protection that sourcekitd provided, but: - The parser is a much less complicated system than the rest of the compiler pipeline, and given the additional quality assurance tools that we have, like the amazing [swift stress tester](https://github.com/apple/swift-stress-tester), I believe we can be confident that we will protect SwiftSyntax against crashing bugs. - Moving SwiftSyntax off sourcekitd has an additional upside beyond performance. A long standing desire of us was to isolate the syntactic functionality from the sourcekitd crashes (a typechecker crash should have no effect on syntactic functionality), and this approach allows us to provide this. Given how critical the performance of SwiftSyntax is for allowing it to replace all uses of the legacy syntactic request, I believe this approach is the way forward in order to achieve this goal. ---
**Author:** ![Jon\_Shier](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/jon_shier/32/921_2.png) [@Jon\_Shier](https://forums.swift.org/u/Jon_Shier)\ **Post date:** [December 2, 2018, 1:09am UTC](https://forums.swift.org/t/speeding-up-swiftsyntax-by-using-the-parser-directly/18493/2 "2018-12-02T01:09:41Z")
As an outsider who isn't involved in this process at all: When SwiftSyntax was introduced, I thought I remembered a future direction of integrating it into the compiler pipeline so it would become the first step, and its output could be shared between the compiler and other tools which need to consume the high fidelity output (like the LS). Is that not the case, or is that just a long long term goal? As a Swift user, yes please! Syntax highlighting breaking because of `sourcekitd` crashes are super annoying, especially since Xcode's "Report a bug" button is cruel joke. Highlighting flashes because the entire file has to be highlighted at once are also super annoying. Flashes while editing code are super super annoying. Anything that improves this situation would be great. On a related note, when Swift LSP support was announced, there was much rejoicing, aside from the comments that, since it would still use `sourcekitd`, every IDE would now suffer from the same issues Xcode sees. So this would help general developer confidence as well. ---
**Author:** ![akyrtzi](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/akyrtzi/32/4963_2.png) [@akyrtzi](https://forums.swift.org/u/akyrtzi)\ **Post date:** [December 2, 2018, 2:05am UTC](https://forums.swift.org/t/speeding-up-swiftsyntax-by-using-the-parser-directly/18493/3 "2018-12-02T02:05:49Z")
> [@Jon\_Shier](#): > > Is that not the case, or is that just a long long term goal? It's a long term goal. ---
**Author:** ![krzyzanowskim](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/krzyzanowskim/32/1282_2.png) [@krzyzanowskim](https://forums.swift.org/u/krzyzanowskim)\ **Post date:** [December 3, 2018, 6:26pm UTC](https://forums.swift.org/t/speeding-up-swiftsyntax-by-using-the-parser-directly/18493/4 "2018-12-03T18:26:57Z")
are you saying that `sourcekitd` "parser" is the slowest possible approach. Slower than playing with JSON ? ---
**Author:** ![akyrtzi](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/akyrtzi/32/4963_2.png) [@akyrtzi](https://forums.swift.org/u/akyrtzi)\ **Post date:** [December 3, 2018, 6:30pm UTC](https://forums.swift.org/t/speeding-up-swiftsyntax-by-using-the-parser-directly/18493/5 "2018-12-03T18:30:17Z")
> [@krzyzanowskim](#): > > are you saying that `sourcekitd` "parser" is the slowest possible approach. Slower than playing with JSON ? No, JSON is multiple times slower than the sourcekitd approach, I just didn't bother taking specific measurements for it, we know this already. ---
**Author:** ![krzyzanowskim](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/krzyzanowskim/32/1282_2.png) [@krzyzanowskim](https://forums.swift.org/u/krzyzanowskim)\ **Post date:** [December 3, 2018, 6:32pm UTC](https://forums.swift.org/t/speeding-up-swiftsyntax-by-using-the-parser-directly/18493/6 "2018-12-03T18:32:56Z")
What is "legacy syntactic request" ? it's faster than "sourcekitd" - I found this part intriguing. ---
**Author:** ![akyrtzi](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/akyrtzi/32/4963_2.png) [@akyrtzi](https://forums.swift.org/u/akyrtzi)\ **Post date:** [December 3, 2018, 6:40pm UTC](https://forums.swift.org/t/speeding-up-swiftsyntax-by-using-the-parser-directly/18493/7 "2018-12-03T18:40:39Z")
It's the functionality for opening a source document and getting a 'syntax-map' (flat array used for syntactic highlighting) and a 'document structure' (tree describing the structure of the code). Check examples for [syntax-map](https://github.com/apple/swift/blob/master/test/SourceKit/SyntaxMapData/syntaxmap.swift.response) and [document-structure](https://github.com/apple/swift/blob/master/test/SourceKit/DocumentStructure/structure.swift.response). ---
**Author:** ![akyrtzi](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/akyrtzi/32/4963_2.png) [@akyrtzi](https://forums.swift.org/u/akyrtzi)\ **Post date:** [January 10, 2019, 8:36pm UTC](https://forums.swift.org/t/speeding-up-swiftsyntax-by-using-the-parser-directly/18493/8 "2019-01-10T20:36:52Z")
An update on this: we started incorporating related changes, like [refactoring how swift syntax parsing works](https://github.com/apple/swift/pull/21368) in order to [slot in a C API for it](https://github.com/apple/swift/pull/21762). There are plenty more changes coming in, and we are also looking into the SwiftSyntax side to make it more efficient to create and visit the tree. Such changes may be source breaking as well. Given that these are disruptive changes and that we'd like to have some time internally to "bake" the changes before making them available via a released toolchain, we are not aiming to have this available for the swift-5.0 release, but it will be part of the next release. ---
**Author:** ![jpsim](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/jpsim/32/3390_2.png) [@jpsim](https://forums.swift.org/u/jpsim)\ **Post date:** [January 11, 2019, 12:07am UTC](https://forums.swift.org/t/speeding-up-swiftsyntax-by-using-the-parser-directly/18493/9 "2019-01-11T00:07:46Z")
This is great news Argyrios. I really appreciate the focus on performance here! ---
**Author:** ![aemino](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/aemino/32/5291_2.png) [@aemino](https://forums.swift.org/u/aemino)\ **Post date:** [January 11, 2019, 3:40am UTC](https://forums.swift.org/t/speeding-up-swiftsyntax-by-using-the-parser-directly/18493/10 "2019-01-11T03:40:10Z")
> [@akyrtzi](#): > > The only potential downside of this approach is that we are losing the crash protection that sourcekitd provided My only worry is this potential downside -- the last time I used Rust, which has the RLS (Rust Language Server), I encountered a lot of issues with the language server primarily because it is very tightly coupled with the Rust compiler infrastructure; this led to crashes or situations where analysis broke down due to an error in the code which would have been easier to fix should analysis have been available. Ultimately, I hope that providing SwiftSyntax direct access to the Swift parser doesn't result in scenarios where static analysis is kneecapped by issues that would be important at compile-time, but are less important while code is being developed. ---
**Author:** ![akyrtzi](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/akyrtzi/32/4963_2.png) [@akyrtzi](https://forums.swift.org/u/akyrtzi)\ **Post date:** [January 11, 2019, 10:22pm UTC](https://forums.swift.org/t/speeding-up-swiftsyntax-by-using-the-parser-directly/18493/11 "2019-01-11T22:22:43Z")
> [@aemino](#): > > Ultimately, I hope that providing SwiftSyntax direct access to the Swift parser doesn't result in scenarios where static analysis is kneecapped by issues that would be important at compile-time, but are less important while code is being developed. Not quite sure I'm following but I don't think what you describe matches what we intend to do. SwiftSyntax will be used only for syntactic processing, and any semantic analysis, like typechecking, will stay behind the sourcekitd process barrier. ---
**Author:** ![inamiy](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/inamiy/32/45098_2.png) [@inamiy](https://forums.swift.org/u/inamiy)\ **Post date:** [January 14, 2019, 7:59am UTC](https://forums.swift.org/t/speeding-up-swiftsyntax-by-using-the-parser-directly/18493/12 "2019-01-14T07:59:44Z")
FYI here are some experiments I made using 1. SwiftLang + ByteTree and 2. newly-released direct-parser: 1. Add `RunSwift5Command` (SwiftLang + ByteTree, experiment) by inamiy 路 Pull Request #13 路 inamiy/SwiftRewriter [https://github.com/inamiy/SwiftRewriter/pull/13](https://github.com/inamiy/SwiftRewriter/pull/13) 2. Update toolchain to `PR-21772-166` & use direct-parser by inamiy 路 Pull Request #14 路 inamiy/SwiftRewriter [https://github.com/inamiy/SwiftRewriter/pull/14](https://github.com/inamiy/SwiftRewriter/pull/14) ---
**Author:** ![inamiy](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/inamiy/32/45098_2.png) [@inamiy](https://forums.swift.org/u/inamiy)\ **Post date:** [March 28, 2019, 5:19am UTC](https://forums.swift.org/t/speeding-up-swiftsyntax-by-using-the-parser-directly/18493/13 "2019-03-28T05:19:52Z")
> Add `RunSwift5Command` (SwiftLang + ByteTree, experiment) by inamiy 路 Pull Request #13 路 inamiy/SwiftRewriter > [https://github.com/inamiy/SwiftRewriter/pull/13](https://github.com/inamiy/SwiftRewriter/pull/13) @akyrtzi @Xi_Ge Is `libswiftSwiftLang.dylib` not available in Xcode 10.2 toolchain? I thought we could at least try ByteTree for faster parsing than in Swift 4.2. ---
**Author:** ![akyrtzi](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/akyrtzi/32/4963_2.png) [@akyrtzi](https://forums.swift.org/u/akyrtzi)\ **Post date:** [March 28, 2019, 3:48pm UTC](https://forums.swift.org/t/speeding-up-swiftsyntax-by-using-the-parser-directly/18493/14 "2019-03-28T15:48:32Z")
We didn't want to put it in the toolchain because we intend at some point to move it out of the Swift repo and into its own SwiftPM package, like SwiftSyntax. Maybe the most convenient thing would be to copy the relevant swift files in your repo, if you want to use them, there's only a few number of them.