GSoC Swift compiler integration with external tools

Hello. My name is Kazutaka Homma. I’m a senior at UC San Diego and I’m very excited to apply Google Summer of Code under Swift!
Currently I’m interested in this task; Swift compiler integration with external tools.
How can I get started?

2 Likes

Hey Kazutaka,
Thank you for being interested in this project!

It largely depends on what you currently know about Swift compiler project.

  • Have you already checked out and built the compiler?
  • Have you already made any contribution to the project?
  • Have you already tried latest libSyntax APIs?

If you haven't done anything above, I recommend you to start them now.
Fixing StarterBugs on JIRA is a good start point. Experiencing it would be a good way to you to getting some ideas about the project.

As a side note, Here's a related discussion about this specific task:

2 Likes

Thank you Rintaro!

I have not done them yet, so I will start working on them right now and also look into StarterBugs!

1 Like

Great!
See also https://swift.org/contributing/
If you have any questions about getting started to contributing, feel free to open a new thread in the forum or even DM me:)

Let’s make this thread to be for questions/discussion about this specific task.

1 Like

Thank you for being supportive!

I’m thinking to work on this bug but wanted to make sure if I can just assign myself to it or is there a place to ask for that permission?

Just go ahead and assign yourself 👍

Hello, Rintaro
Thank you for the review for my PR btw! I’m working on it.

Also, I need to work on the proposal and then I’ve got a couple of questions for the project.

Description
Add a compiler option that provides the path to an external tool for the compiler to execute and communicate with. The communication could be done via stdin/stdout using a JSON format. The compiler should pass the compiler arguments and the libSyntax tree of the currently compiling source file, allowing the tool to return custom diagnostics that the compiler includes along with the rest of the compiler diagnostics.

From the project description, I understood an external tool will have an access to libSyntax, compiler arguments, and ability to add additional diagnostic. The access does not include any other compiler feature. Is that correct?

The external tool itself should:

  1. Receive a libSyntax tree and compiler arguments form, via STDIN using JSON format
  2. Return diagnostics via STDOUT using JSON format

Also, external tools should be able to be written in Swift. So I expect you to design/implement a Swift framework using SwiftSyntax.

Thank you @rintaro!

I got it. Thank you!

This swift framework that I’m expected to implement is for reconstructing libSyntax tree from JSON format and what else does it need to do?

I was also looking at another post by you [1]

From this, I assumed I used the same seializer and deserializer for my task. is that correct?

To be accurate, the current Parser generates both AST and libSyntax tree. But I guess it’s difficult to use the current libSyntax tree generation functionality for this task. Currently, it’s like a by-product of AST parsing.
This means his task would change the libSyntax to be generaeted

Should I worry about how the libSyntax tree output would change due to his change?

[1] [GSOC]Integration of libSyntax with the rest of the compiler pipeline

The existing SwiftSyntax library already has functionality for a) deserializing libSyntax tree JSON and b) DiagnosticsEngine which is able to output JSON. What you have to do is designing a framework to make a command line tool using this SwiftSyntax library. (handing command line arguments, read from STDIN, etc.)

Basically yes. In addition you have to implement:

  • Serializer combining compiler arguments and libSyntax tree.
  • Deserializer for diagnostics emitted by the external tools.

I don’t think so. Although you should at least know how libSyntax trees are serialized, as I said, deserialization of libSyntax tree is implemented in SwiftSyntax library. You can just use it without worrying about how they are serialized for implementing external tools framework.

Thank you @rintaro

I think I start getting the idea!

Is it correct that compiler interact with framework via stdin/stdout and external tool just interact with compiler through the framework?
Compiler ⇄ Framework ⇄ External Tool

Also, should the libSyntax tree be outputted incrementally? or can it wait until it finish parsing the whole clause?

Yeah, something like that. For example, a tool author write external tool like this.

import SwiftToolKit
import SwiftSyntax

SwiftTool.run(Process.arguments) { ctxt in
  let compilerArguments: [String] = ctxt.compilerArguments
  let syntaxTree: SourcefileSyntax = ctxt.sourceFileSyntax
  let diag: DiagnosticEngine = ctxt.diagnosticEngine

  // ... walk into 'syntaxTree'. Lint check, whatever ...
  // diag.diagnose(...) to emit diagnostic messages.
  // After this function call finished, the framework should serialize
  // all diagnostic messages to a JSON, and output it to STDOUT.
}

and compile it.

To use this tool, a user can invoke the compiler with the path to the compiled tool (let’s say -external-tool):

$ swiftc -external-tool path/to/compiled-tool -o ./myprogram main.swift

In this way,

  1. The compiler parse the source file, build syntax tree, invoke path/to/compiled-tool
  2. path/to/compiled-tool receive the syntax tree via STDIN, emit diagnose messages to STDOUT
  3. The compiler receives the messages, print out them as if they are messages the compiler itself emits.
$ swiftc -external-tool path/to/compiled-tool -o ./myprogram main.swift
main.swift:204:7: warning: hey, non-CamelCase type name violates our policy!
   class my_program {
         ^~~~~~~~~~
         MyProgram

You mean, if the source file has multiple decls at top-level, should the compiler invoke the external tool multiple times? Interesting idea! Although I don’t think it would be needed in the initial implementation, feel free to propose it if you want! At least, it should be optional, say -external-tool-incremental.

Hey Ritaro,
I recently know about GSoC Swift compiler integration with external tools. I really excited on applying for this project on GSoC.

I don’t have any experience with Swift compiler project, so I will start with the StarterBugs on JIRA…

Thank you

Just a reminder that all GSoC proposals need to be filed by March 27.

1 Like

Thank you for your reminder!

Hi, @Rintaro
Thank you for your detailed explanation. That helped me understand the project a lot!
I’m also sorry that I could not reply you more quickly. I have had final exams in the last week but it’s done now.

I get that. Yes, I felt that would be too complicated.

I was thinking in which part of compiler the code for each of the following tasks should be placed.

  • compiler option for the external tool executable path
  • execute an external tool
  • write the external tool’s stdin
  • read the external tool’s stdout
  • execute diagnosis from the external tool

My best guess so far is Swift Frontend but I’m not so confident about this. I’ve tried to move up the function call hierarchy from Syntax class but I reached no caller before reaching where the completed libSyntax tree is accessible.
Although I might just need more of digging into the code base and understand how each component is tied together, how do you generally figure out where to modify or add code when you intend to add a new feature especially in such a huge code base?

Hi Kazutaka,

I also guess that it’ll be implemented in libFrontend. And also Driver should pass " external tool executable path" to Frontend.

Good question :slight_smile: . My answer: before you get familiar with the codebase, don’t hesitate to ask someone.

okay, glad that I was not terribly off :]

This is such a relief for me! I’ve kept thinking I’m not good enough to understand the codebase but now I feel much better.

Hi @rintaro
I’ve just submitted my final draft of my proposal! I’m really sorry that I could not have submitted it early enough for you to review it but I hope I get selected and get able to work with you!
I will continue working on the PR too!