GSoC Swift compiler integration with external tools

[Kazutaka333 (Kazutaka Homma)]

Hello. My name is Kazutaka Homma. I'm a senior at UC San Diego and I'm very excited to apply Google Summer of Code under Swift!
Currently I'm interested in this task; Swift compiler integration with external tools.
How can I get started?

[rintaro (Rintaro Ishizaki)]

Hey Kazutaka,
Thank you for being interested in this project!

@Kazutaka333

How can I get started?

It largely depends on what you currently know about Swift compiler project.

• Have you already checked out and built the compiler?
• Have you already made any contribution to the project?
• Have you already tried latest libSyntax APIs?

If you haven't done anything above, I recommend you to start them now.
Fixing StarterBugs on JIRA is a good start point. Experiencing it would be a good way to you to getting some ideas about the project.

As a side note, Here's a related discussion about this specific task:

Swift compiler plugins

I’ve recently been looking into implementing a lint checker for Swift code in my project. I’ve used SwiftLint[1] so far, but I’d love to build something that integrates more closely with Xcode: only check what is being rebuild, provide Fix-Its,… From what I’ve found there’s no equivalent of Clang Plugins[2] in the Swift compiler. Is this something interesting or already planned? Should it go through a swift-evolution proposal? [1] GitHub - realm/SwiftLint: A tool to enforce Swift style and c…

[Kazutaka333 (Kazutaka Homma)]

Thank you Rintaro!

I have not done them yet, so I will start working on them right now and also look into StarterBugs!

[rintaro (Rintaro Ishizaki)]

Great!
See also Swift.org - Contributing
If you have any questions about getting started to contributing, feel free to open a new thread in the forum or even DM me:)

Let's make this thread to be for questions/discussion about this specific task.

[Kazutaka333 (Kazutaka Homma)]

Thank you for being supportive!

I'm thinking to work on this bug but wanted to make sure if I can just assign myself to it or is there a place to ask for that permission?

[rintaro (Rintaro Ishizaki)]

Just go ahead and assign yourself

[Kazutaka333 (Kazutaka Homma)]

Hello, Rintaro
Thank you for the review for my PR btw! I'm working on it.

Also, I need to work on the proposal and then I've got a couple of questions for the project.

Description
Add a compiler option that provides the path to an external tool for the compiler to execute and communicate with. The communication could be done via stdin/stdout using a JSON format. The compiler should pass the compiler arguments and the libSyntax tree of the currently compiling source file, allowing the tool to return custom diagnostics that the compiler includes along with the rest of the compiler diagnostics.

From the project description, I understood an external tool will have an access to libSyntax, compiler arguments, and ability to add additional diagnostic. The access does not include any other compiler feature. Is that correct?

[rintaro (Rintaro Ishizaki)]

The external tool itself should:

1. Receive a libSyntax tree and compiler arguments form, via STDIN using JSON format
2. Return diagnostics via STDOUT using JSON format

Also, external tools should be able to be written in Swift. So I expect you to design/implement a Swift framework using SwiftSyntax.

[Kazutaka333 (Kazutaka Homma)]

Thank you @rintaro!

@rintaro

Receive a libSyntax tree and compiler arguments form, via STDIN using JSON format
Return diagnostics via STDOUT using JSON format

I got it. Thank you!

@rintaro

Also, external tools should be able to be written in Swift. So I expect you to design/implement a Swift framework using SwiftSyntax.

This swift framework that I'm expected to implement is for reconstructing libSyntax tree from JSON format and what else does it need to do?

I was also looking at another post by you [1]

@rintaro

We already have a serializer; try swift -frontend -emit-syntax source.swift with the latest master build. For the deserializer, we are able to use LLVM YAML parser.

From this, I assumed I used the same seializer and deserializer for my task. is that correct?

To be accurate, the current Parser generates both AST and libSyntax tree. But I guess it’s difficult to use the current libSyntax tree generation functionality for this task. Currently, it’s like a by-product of AST parsing.
This means his task would change the libSyntax to be generaeted

Should I worry about how the libSyntax tree output would change due to his change?

[1] [GSOC]Integration of libSyntax with the rest of the compiler pipeline - #4 by rintaro

[rintaro (Rintaro Ishizaki)]

@Kazutaka333

This swift framework that I’m expected to implement is for reconstructing libSyntax tree from JSON format and what else does it need to do?

The existing SwiftSyntax library already has functionality for a) deserializing libSyntax tree JSON and b) DiagnosticsEngine which is able to output JSON. What you have to do is designing a framework to make a command line tool using this SwiftSyntax library. (handing command line arguments, read from STDIN, etc.)

@Kazutaka333

From this, I assumed I used the same seializer and deserializer for my task. is that correct?

Basically yes. In addition you have to implement:

• Serializer combining compiler arguments and libSyntax tree.
• Deserializer for diagnostics emitted by the external tools.

@Kazutaka333

Should I worry about how the libSyntax tree output would change due to his change?

I don't think so. Although you should at least know how libSyntax trees are serialized, as I said, deserialization of libSyntax tree is implemented in SwiftSyntax library. You can just use it without worrying about how they are serialized for implementing external tools framework.

[Kazutaka333 (Kazutaka Homma)]

Thank you @rintaro

I think I start getting the idea!

Is it correct that compiler interact with framework via stdin/stdout and external tool just interact with compiler through the framework?
Compiler ⇄ Framework ⇄ External Tool

Also, should the libSyntax tree be outputted incrementally? or can it wait until it finish parsing the whole clause?

[rintaro (Rintaro Ishizaki)]

@Kazutaka333

Compiler ⇄ Framework ⇄ External Tool

Yeah, something like that. For example, a tool author write external tool like this.

import SwiftToolKit
import SwiftSyntax

SwiftTool.run(Process.arguments) { ctxt in
  let compilerArguments: [String] = ctxt.compilerArguments
  let syntaxTree: SourcefileSyntax = ctxt.sourceFileSyntax
  let diag: DiagnosticEngine = ctxt.diagnosticEngine

  // ... walk into 'syntaxTree'. Lint check, whatever ...
  // diag.diagnose(...) to emit diagnostic messages.
  // After this function call finished, the framework should serialize
  // all diagnostic messages to a JSON, and output it to STDOUT.
}

and compile it.

To use this tool, a user can invoke the compiler with the path to the compiled tool (let's say -external-tool):

$ swiftc -external-tool path/to/compiled-tool -o ./myprogram main.swift

In this way,

1. The compiler parse the source file, build syntax tree, invoke path/to/compiled-tool
2. path/to/compiled-tool receive the syntax tree via STDIN, emit diagnose messages to STDOUT
3. The compiler receives the messages, print out them as if they are messages the compiler itself emits.

$ swiftc -external-tool path/to/compiled-tool -o ./myprogram main.swift
main.swift:204:7: warning: hey, non-CamelCase type name violates our policy!
   class my_program {
         ^~~~~~~~~~
         MyProgram

@Kazutaka333

Also, should the libSyntax tree be outputted incrementally? or can it wait until it finish parsing the whole clause?

You mean, if the source file has multiple decls at top-level, should the compiler invoke the external tool multiple times? Interesting idea! Although I don't think it would be needed in the initial implementation, feel free to propose it if you want! At least, it should be optional, say -external-tool-incremental.

[nockzblack (Fernando Benavides)]

Hey Ritaro,
I recently know about GSoC Swift compiler integration with external tools. I really excited on applying for this project on GSoC.

I don't have any experience with Swift compiler project, so I will start with the StarterBugs on JIRA...

Thank you

[tkremenek (Ted Kremenek)]

Just a reminder that all GSoC proposals need to be filed by March 27.

[Kazutaka333 (Kazutaka Homma)]

Thank you for your reminder!

[Kazutaka333 (Kazutaka Homma)]

Hi, @Rintaro
Thank you for your detailed explanation. That helped me understand the project a lot!
I'm also sorry that I could not reply you more quickly. I have had final exams in the last week but it's done now.

@rintaro

You mean, if the source file has multiple decls at top-level, should the compiler invoke the external tool multiple times? Interesting idea! Although I don’t think it would be needed in the initial implementation, feel free to propose it if you want! At least, it should be optional, say -external-tool-incremental.

I get that. Yes, I felt that would be too complicated.

I was thinking in which part of compiler the code for each of the following tasks should be placed.

• compiler option for the external tool executable path
• execute an external tool
• write the external tool's stdin
• read the external tool's stdout
• execute diagnosis from the external tool

My best guess so far is Swift Frontend but I'm not so confident about this. I've tried to move up the function call hierarchy from Syntax class but I reached no caller before reaching where the completed libSyntax tree is accessible.
Although I might just need more of digging into the code base and understand how each component is tied together, how do you generally figure out where to modify or add code when you intend to add a new feature especially in such a huge code base?

[rintaro (Rintaro Ishizaki)]

Hi Kazutaka,

@Kazutaka333

My best guess so far is Swift Frontend but I’m not so confident about this. I’ve tried to move up the function call hierarchy from Syntax class but I reached no caller before reaching where the completed libSyntax tree is accessible.

I also guess that it'll be implemented in libFrontend. And also Driver should pass " external tool executable path" to Frontend.

@Kazutaka333

how do you generally figure out where to modify or add code when you intend to add a new feature especially in such a huge code base?

Good question . My answer: before you get familiar with the codebase, don't hesitate to ask someone.

[Kazutaka333 (Kazutaka Homma)]

@rintaro

I also guess that it’ll be implemented in libFrontend. And also Driver should pass " external tool executable path" to Frontend.

okay, glad that I was not terribly off :]

@rintaro

Good question . My answer: before you get familiar with the codebase, don’t hesitate to ask someone.

This is such a relief for me! I've kept thinking I'm not good enough to understand the codebase but now I feel much better.

[Kazutaka333 (Kazutaka Homma)]

Hi @rintaro
I've just submitted my final draft of my proposal! I'm really sorry that I could not have submitted it early enough for you to review it but I hope I get selected and get able to work with you!
I will continue working on the PR too!

[Snow]

Hello Rintaro, my project also needs a compile-time lint checker for swift code, does swift currently supports the external tools mentioned in this post (similar to the clang plugin)?