Formatting technology for SourceKit-LSP and other tools

Hey all,

Myself, @blangmuir, @Xi_Ge, and others who spend a lot of their time working on SourceKit/SourceKit-LSP and SwiftSyntax have been thinking about bringing formatting technology to SourceKit-LSP. Formatting is integral to the experience of editing source code which why it is included in the LSP specification. This functionality, however, is currently missing from SourceKit-LSP. We want to provide users a great out-of-the-box experience of using SourceKit-LSP, across all editors and platforms, and first-class support for formatting is a critical piece of the experience.

For SourceKit-LSP we'd like to have formatting functionality that is:

  1. Fast and capable of real-time, "as you type", performance
  2. Be robust and work even in the presence of invalid code
  3. Work in contexts that may only provide a string of Swift source code for context such as snippets in code-review

For these workflows doing full semantic analysis is either too expensive or impossible due to lack of context (e.g. missing compiler arguments and inability to import the SDK or other dependencies). Fortunately, the Swift grammar was designed to be parsed without semantic analysis to enable tooling like this. Combining these requirements and factors, our belief is that the formatting technology should focus on using the syntactic structure of code without doing semantic analysis.

Looking at higher level goals, the formatting technology for SourceKit-LSP should aim to provide the building blocks for doing code format transformations that can be used for a variety of tools. To provide these building blocks but also not pull in all of SourceKit-LSP, we believe that the formatting technology should be provided as a reusable SwiftPM package that SourceKit-LSP itself uses. This layering would allow other tools to build their own formatting pipelines or leverage pieces that help them focus on the tasks that are most relevant to those tools. For example, tools like SwiftLint (which is doing a lot more than just whitespace transformations) can delegate some tasks to this infrastructure and focus more on adding additional rules and functionality. Further, the formatting technology could be used by code-generation tools that would like to have available convenient APIs to format the resulting code. By using the formatting technology from the package, these kinds of tools can be liberated from worrying about the particulars of handling a Swift grammar construct unless it is relevant for some particular rule.

Following the principle of building foundational pieces that we can then utilize to build other functionality, we also believe that the formatting technology should be built on top of SwiftSyntax. SwiftSyntax is directly integrated with the Swift parser, and there are ongoing efforts focused to keep improving its performance and the convenience of using SwiftSyntax for building Swift tools that need to understand the syntactic structure of Swift code. For example, SwiftSyntax has performance features like incremental re-parsing so that a live editor will do only the minimum work necessary to get an updated syntax tree after small user edits. Further, any change in the Swift grammar is going to be reflected in the SwiftSyntax APIs, either from the direct implementation of a SE proposal or at least soon after a proposal is accepted.

Given the above goals and motivations I'm excited about Google's swift-format project. The work that has gone into the swift-format tool on Google's swift repository aligns with many of the technical goals outlined here, and that team, led by @allevato, has expressed interest in contributing that work back to be part of this new formatting package to help achieve the goals outlined above. We will create a new swift.org repository and the Google team will be providing pull requests to contribute their code to that repository. We'd like everyone who is interested to participate in discussions in reviewing and incorporating such contributions and any other new work to build the new formatting technology.

I'm super excited about getting formatting functionality into swift.org and SourceKit-LSP and would love to hear your feedback about the contributed code and also how you could utilize the technology and what use-cases and workflows are most important to you. Help us with our continuous efforts to build awesome Swift tools!

30 Likes

Thanks Argyrios! I'm excited to contribute this upstream and have it be part of the Swift project.

I'd also like to give credit to the folks who contributed way more code than I did and without whom it wouldn't have been possible: @dabelknap, @harlanhaskins, @abl, @LaurenWhite, and @atamez31.

11 Likes

Would this perhaps be a candidate directly being executed by Xcode internally to keep our source files formatted appropriately? I know that's an orthogonal concern, but that would certainly make me swoon!

2 Likes

How does this relate to SE‐0250 that was left hanging?

See Ted's last post on the thread, this is about "technology to support formatting", as mentioned in that post.

1 Like

That’s an amazing news! Swift-format from Google (IMHO) has proved to be already solid and fast enough. It’s great to see that we can merge efforts in order to improve it. Bravo!

It would be nice if we integrate this in swift cli, maybe under an experimental flag, so that the community can play with it, without compiling or installing other components, what’s your view about this?

We can have it in the master development snapshots to experiment with.

1 Like

When we first announced SourceKit-LSP we mentioned that we intend for Xcode to use it as well, in which case SourceKit-LSP's formatting support would come along with it :slightly_smiling_face:.

4 Likes

We've created a new swift-format repository and @allevato created the PR to populate it! :tada:
There's also a new forum category for discussions related to swift-format.

I'd love to see your feedback either on the PR or the forums!

10 Likes

There should be ways to import the commit history from google/swift. I'm wondering if squashing everything into a single commit is a deliberate choice :thinking:

It's deliberate.

1 Like

What are the reasons?

It's better to start the repo with the proper licensing information as included in all the swift.org projects.

It appears late to change approach now, but fwiw the commits could have been imported, with history, on top of the properly initialized & licensed new repo.

1 Like