GSoC libSyntax-based indentation mechanism

Hello everyone! My name is Dhruv Garg. I'm a 2nd year undergraduate student from India and I am very excited to apply for GSoC under Swift.

I am interested in the following task : libSyntax-based indentation mechanism.
Before going into the details of libSyntax, I would like to have a good understanding of the current indentation mechanism in Swift.
How can I get started?

2 Likes

Welcome Dhruv! Excited to see that you're interested!

@akyrtzi, @Xi_Ge, and @rintaro are good folks to reach out to on this.

Hi Dhruv,
I'm glad you're interested in learning more about the indentation mechanism of Swift. Currently, based on Swift AST instead of Swift syntax tree, our indentation is implemented in swift/lib/IDE/Formatting.cpp. There're multiple areas we'd like to improve about this mechanism:

  1. To support multi-line indentation: Our current implementation only allows single-line indentation. We'd like to support multi-line input or whole-file re-formatting in one transaction for performance enhancement.
  2. To improve configurability: We'd like to allow more detailed configuration specified from users, a classic example is whether to indent case statements inside a switch statement.
  3. To use libSyntax/Swift for implementation: @harlanhaskins is actively working on using libSyntax/Swift to re-implement the formatting logic. We should explore the marriage between this new implementation and our IDE service layer swift/tools/SourceKit.

Please let us know which area fits your interest better.
Xi

Hi Xi and Harlan,

Thank you for your prompt responses. The reason I want to work on indentation / formatting code is because it makes the life of a developer a lot simpler. The code easier to read, understand, modify, maintain and enhance.

While exploring about libSyntax and Swift syntax tree a week ago, i watched a video "Swift tools with libSyntax". Only to realize now that it was by @harlanhaskins himself!! Pretty cool! I enjoyed the video and liked the work he is doing.

@Xi_Ge The 1st and 3rd, both interest me.

  1. Would I be right to say that multi-line indentation mandates working with Swift AST and swift/lib/IDE/Formatting.cpp and not libSyntax? That said, does this apply to user-configurable indentation too, as stated in point 2?
  2. I really like the work being done with libSyntax to re-implement formatting. It would be nice for it to be integrated with the IDE service layer. However, I wish to explore more on this. It would be great if more light can be thrown onto this topic.

Dhruv

1 Like

Yeah, we all have the great honor to work with @harlanhaskins : )

That's a really good question. The answer depends on whether we can integrate the libSyntax formatter with SourceKit in timely manner. If we're sure to integrate them successfully, there can be a strong motivation to replace AST-based formatter with libSyntax-based formatter entirely; and in that case, we can implement the multi-line formatting logic in the libSyntax formatter alone.

Exploring SourceKit will also be a good idea! I suggest looking into our Swift wrapper of SourceKit service (it's called swift-lang). Like Swift syntax, you can find this module in open source toolchain. A starter task can be: using swift-lang to invoke SourceKit to format a given Swift source file, say test.swift. A code example to use swift-lang can be found here.

Hello Xi,

Apologies for the late reply.

It is pretty clear from your response that the integration of libSyntax formatter with SourceKit lies at the heart of replacing the AST-based formatter with the libSyntax formatter. If we are sure about the benefits of using libSyntax formatter over the current AST-formatter, then I think that our first priority should definitely be to integrate the libSyntax formatter with the Sourcekit service.

I am exploring the Sourcekit and working on the starter task. Will get back to you soon on this.
However, there is one thing that I would like to know. In some codes it is mentioned that it requires Mac OS X, while I am working on Ubuntu 16.04. Is this a cause of major concern?

Dhruv

Yes, I believe so as well! In your project idea proposal, please mention that your plan of integrating standalone swift-formatter with SourceKit as the first priority.

hmm, unfortunately, this is an issue. Currently, our Swift wrapper on SourceKit only builds in Mac OS X. However, you can work on SourceKit without using the Swift wrapper. A starter task on SourceKit can be "Allow SourceKitd client to specify whether indenting case statement in switch statements". You basically need to add one more flag in here, and start using it in the formatting implementation. Please let me know if you have encountered any blockers.

Hello Xi,

Alright! That's great! In my proposal, I will mention the integration of the libSyntax based formatter with SourceKit as the first priority.

Okay, will work on SourceKit without the Swift wrapper. In any case, I'm sure that we will be able to find workarounds for the issues that might come up.

For this task, the following is my thought process. Please correct me if wrong.

Currently in swift/IDE/Formatting.h,

bool IndentSwitchCase = false;

Thus, by default, indenting at the case-level is disabled.

To allow the SourceKitd client to specify whether to indent case in switch statements, I need to add a flag to make :

bool IndentSwitchCase = true;

when case-level statement needs to be indented. After this, I need to add code / make required changes in swift/IDE/formatting.cpp for parts where the switch-case statements are being operated upon.

I see that struct CodeFormatOptions is being used across various files, like:

swift/IDE/formatting.cpp, SwiftLang/SwiftEditor.cpp, SwiftLang/SwiftLangSupport.h

However, I am trying to figure out where the format option for switch-case is set and how it will be passed on to swift/IDE/Formatting.h from the IDE layer.

Hi Everyone, I am Kushal. I was going through the project ideas list and found this really interesting. I am currently learning and building a compiler as part of a course in our university. I want to contribute to this project. Exploring libSyntax right now. Gotta say it's an amazing tool.
I am primarily interested in doing this and also think that's the primary goal here. (as per my understanding from this thread)

I'm right now going through SourceKit and testing it out.
I'd like to know if there are any additional resources that are helpful for this.

About me:
I am a 3rd year under-graduate at IIT Guwahati, India. I am primarily a C/C++ developer working my way into swift. I've completed a project for 'The FreeType Org' under the GSoC 2017 program.

Thank You.

Hi Kushal,
Glad to hear that! When you submit a GSoC proposal for review, please include any ideas you have for using libSyntax, or any enhancement you have in mind for the syntax API itself. For example, we can explore the educational value of using libSyntax APIs like what you mentioned for your course.

One thing I've mentioned above is to use the Swift wrapper for sourcekit (currently called swiftLang) to experiment SourceKitd functionalities. For instance, we can use cursor-info request in swiftLang to give libSyntax semantic information.

Nice to meet you again, Kusal! I cannot wait to see what you can do for Swift as an open source contributor.

1 Like

Sounds great!

Sorry @dhruvsgarg , it was my bad that I've not realized this flag does exist in our codebase. So, instead, we can teach the formatter another flag for configuration. For instance, we can add a flag IndentParameterByColon in format option. When it's true, we align parameters in function-like parameters by ":", e.g.

func foo(first: Int,
        second: Int,
         third: Int) {}

Could you try to add this option? To set this flag, we need to teach sourcekitd; you can find how we set other flags in here.

Hello Xi,

Yes, definitely. I will work on it and get back to you. Thanks for the link!

I'll approach it in the same way :

  1. Go through /SwiftLang/SwiftEditor.cpp
  2. Add a flag IndentParameterByColon in swift/IDE/Formatting.h
  3. Start using the flag in the formatting implementation files like swift/IDE/Formatting.cpp

Also, I wanted to know about the best practice(s) I should apply, to build and test my implementation in swift/lib/IDE and swift/tools/SourceKit.

Since I'm on the Ubuntu platform, I will be using the Ninja build system. The swift/README.md reads as follows:

Once the first build has completed, Ninja can perform fast incremental builds of various products. These incremental builds are a big timesaver when developing and debugging.

Does it mean that the first thing I should is: complete the first build. Only after this, start my implementation and then perform smaller, incremental builds on the two libraries that I will be making changes to?

Since you're on ubuntu, i think the best practice is to test is using swift-formatter. Once you've implemented the option in the formatting engine, you can follow the test here. However, you don't need to implement everything at one shot. I think we can have a WIP (work in progress) PR on github to iterate. By having a PR, we can also use @swift-ci to test your change incrementally.

Hello Xi,

Alright, thanks! I will check it out. Also, would it be right to assume that you are collectively calling swift/lib/IDE, SourceKit/lib/SwiftLang and Sourcekit/tools/sourcekitd as the 'swift formatting engine'?

The work in progress (WIP) PR does make a lot of sense. It would be a really good way to begin getting feedback on something that is still under construction. I will open a WIP: PR in a day or two and it would be great to have your valuable feedback!

Hello everyone,

I have almost completed my proposal and will be uploading it today.

In the meantime, I was trying to interact with SourceKit using SourceKitten. I am trying to obtain the token output and the AST output from the source code. However, Im having a small problem with the installation of sourcekitten. It is described as given below.

OS: Ubuntu 16.04 LTS
Swift: v4.0.3

$ git clone https://github.com/jpsim/SourceKitten.git
[...]
$ cd SourceKitten/
$ swift build

Fetching https://github.com/drmohundro/SWXMLHash.git
Fetching https://github.com/jpsim/Yams.git
Fetching https://github.com/norio-nomura/SourceKit.git
Updating https://github.com/Quick/Quick.git
[....]
Compile CYaml src/reader.c
Compile CYaml src/scanner.c
Compile CYaml src/parser.c
[....]
Compile Swift Module 'SourceKittenFramework' (36 sources)
Compile Swift Module 'sourcekitten' (10 sources)
Linking ./.build/x86_64-unknown-linux/debug/sourcekitten

This is where the SourceKitten installation ends. It does not execute the following
(as shown by @jpsim in swift build errror · Issue #470 · jpsim/SourceKitten · GitHub):

Linking ./.build/x86_64-unknown-linux/debug/SourceKittenPackageTests.xctest
Test Suite 'All tests' started at 2018-01-27 00:02:55.970
[...]
Test Suite 'All tests' passed at 2018-01-27 00:03:09.026
   Executed 42 tests, with 0 failures (0 unexpected) in 13.05 (13.05) seconds

On using "sourcekitten help" I get an error saying sourcekitten : command not found.
I think that the execution is getting terminated as the XCTests are not being run.

I have also upgraded clang from 3.8 to 4.0 and set clang-4.0 as default.

I thought that it would be best to ask developers on the forum before raising the issue with @jpsim.

All help/suggestions would be highly appreciated.