I've made it my goal to be able to contribute to the Swift compiler by the end of 2019.
For some background, most of my software development experience is in iOS. I've never worked on a compiler, and I haven't touched C++ since a data structures and algorithms course I took in college 3 years ago.
Is there a particular part of C++ that would be handy to know? Are there things about compiler development in particular I ought to study?
Thanks!
This may be a good opportunity to expand the (bare) contributor guidelines from the Swift repo?
I am interested too! I downloaded the sources of Swift but I have a hard time getting the whole picture of how everything works, for instance when looking at a bug report, even a beginner one, it's not that easy to know where to start.
Also I don't know how feasible is it to contribute code to Swift as a hobby? Does it take a huge amount of time to understand enough to make small contributions if you never worked on compilers before and aren't a C++ expert?
I've made it my goal to be able to contribute to the Swift compiler by the end of 2019.
It's an admirable goal too. I'd like to help you reach it however I can.
Is there a particular part of C++ that would be handy to know? Are there things about compiler development in particular I ought to study?
C++ is a notoriously dense language that can be a real pain to get a hold on. That said, LLVM-style projects have carved out their own subset of the language, and are very consistent about sticking to that subset. If you want to get a feel for the minutiae, the LLVM Coding Standards document is a great way to get a feel for the aesthetics. If you're looking for a technical start, including programming patterns, data structures, algorithms, and more, look to the LLVM Programmer's Manual.
I learn best by doing, so I'm sparse on book recommendations and such, but I think if you spend enough time with it that you can get up and working with LLVM-style structures pretty quickly.
I will say from personal experience that a formal compilers course or something won't be much help to you here. Resources like The Dragon Book, The Tiger Book, PFPL, etc. that form the foundation of these courses are good for grounding in theory, but they will not help you in practice. These books do not accurately capture the state of real-world compiler development. But if you're just looking for "what is a compiler and how do I put one together", they're each a pretty good answer.
Basically, don't worry about it. Compilers have this reputation for being dense black boxes that they earned over decades of arcana and implementation details leaking out into the broader programming world, or from overly-rigorous college courses leaving a bad taste in people's mouths. It's a terrible misconception that you have to be some kind of wizard to even approach one, and I hope projects like Swift, Rust, and Go can all help to actively challenge it.
Also I don't know how feasible is it to contribute code to Swift as a hobby? Does it take a huge amount of time to understand enough to make small contributions if you never worked on compilers before and aren't a C++ expert?
Just pick a problem and get going. Please feel free to reach out to me personally if you have any questions or need guidance or just wanna talk really! You can get ahold of me through private forum messages or on Twitter usually. My DMs are always open for Swift questions, no matter what.
Finally, a little shameless self-promotion. My interactions with LLVM these days are mostly improving its C API to support Swift bindings. @harlanhaskins has ported parts of LLVM's famous Kaleidoscope tutorial to Swift and has a series of blog posts about it. I'm hoping to update those blog posts and expand them to support LLVM 7.0 and the new JIT in the coming days/weeks/months.
If you're really motivated, I think you can contribute by the end of the of 2018.
I'll start off by saying that different people learn differently than others, so my way of teaching this may not help you, but it may help someone! It's also worth noting that there are many different moving parts of the Swift compiler. By this I mean that there are a lot of different projects working together like the actual compiler, the standard library, foundation, dispatch, xctest, the package manager, the new SourceKit-LSP, and a lot more projects that are all working hard to make Swift the best it can be. A lot of these projects are in Swift itself, so it's really easy to contribute and give back.
Swift Standard Library - https://github.com/apple/swift/tree/master/stdlib/public/core
Swift Package Manager - https://github.com/apple/swift-package-manager
Foundation - https://github.com/apple/swift-corelibs-foundation
SourceKit-LSP - https://github.com/apple/sourcekit-lsp
You can find more projects here - https://github.com/apple
Before I start, I think it's very important that we understand the general pipeline of the compiler. This is useful to understand where new changes or where to fix things should go. In each section, I'll show you an example of what the process is doing, and at the end of each section I'll give you an error that happens in that stage. Please bear in mind that I'm still relatively new to compiler development as well and that mostly everything here is being explained at a high level. Shall we begin?
This is the first part in the compilation process where the compiler breaks down your source code into what we call tokens.
// Source code
let x = 16
// The Lexer turns this into tokens
[kw_let, identifier("x"), equal, integer_literal(16)]
// kw means keyword
The goal of the lexer is to be able to produce this sequence of tokens for the parser to build an AST (Abstract Syntax Tree) from it.
An example error that can happen during this stage:
// error: unterminated string literal
let x = "
^
The lexer wants to make a string_literal token, but notices that there is not a matching "
Lexer.h (Header file) - https://github.com/apple/swift/blob/master/include/swift/Parse/Lexer.h
Lexer.cpp (Implementation) - https://github.com/apple/swift/blob/master/lib/Parse/Lexer.cpp
Lexer Diagnostics (errors, warnings, notes) - https://github.com/apple/swift/blob/master/include/swift/AST/DiagnosticsParse.def#L40-L197
Lexical Analysis - https://en.wikipedia.org/wiki/Lexical_analysis
The Parser takes this tokens and tries build the AST (Abstract Syntax Tree). This is how the compiler actually is able to understand your source code and make sense of it.
// Tokens from lexer
[kw_let, identifier("x"), equal, integer_literal(16)]
// The Parser transforms this into the AST
let x = 16
Here we can see that we actually understand what the parser formed here. At this stage, we can also start adding some flags to the compiler to see some more of how the compiler understands and lowers our source code.
$ swiftc -dump-parse example.swift
This will dump the basic pre-typechecked AST that the parser has built from our source code:
(source_file
(top_level_code_decl
(brace_stmt
(pattern_binding_decl
(pattern_named 'x')
(integer_literal_expr type='<null>' value=16))))
(var_decl "x" type='<null type>' let storage_kind=stored))
We can see our variable declaration for x and we can also see that our integer_literal(16) has turned into an expression for something called a pattern binding declaration. (For now we don't really need to understand what that means to really get the gist of what is happening here. But, a pattern binding declaration is basically binding an expression to a pattern. In this case that pattern is x).
An example error that happen during this stage:
// Tokens produced from lexer for this source code
// [kw_struct, l_brace, r_brace]
// error: expected identifier in struct declaration
struct {}
^
The parser sees this kw_struct token from the lexer and wants to make a struct declaration from it, but the next token is not an identifier to name the structure, but rather the next token is l_brace (left brace).
Parse Headers - https://github.com/apple/swift/tree/master/include/swift/Parse
Parse Implementation files - https://github.com/apple/swift/tree/master/lib/Parse
Parse diagnostics - https://github.com/apple/swift/blob/master/include/swift/AST/DiagnosticsParse.def
Parsing - https://en.wikipedia.org/wiki/Parsing
Abstract Syntax Tree - https://en.wikipedia.org/wiki/Abstract_syntax_tree
Sema is our next step in the compilation process. This includes things like the Type Checker and the Constraint System. There are a lot of goals that this step has to accomplish like assign types to expressions, ensure our structs are conforming to protocols properly, automatically deriving protocol conformance for types, or synthesizing code like the memberwise initializer for structs, and many more.
// AST from the parser
let x = 16
// Sema refined AST
internal let x: Int = 16
I know, it doesn't seem like Sema did much in this step, but there is another flag we can pass to the compiler now!
$ swiftc -dump-ast example.swift
This will dump the typechecked AST.
(source_file
(top_level_code_decl
(brace_stmt
(pattern_binding_decl
(pattern_named type='Int' 'x')
(call_expr implicit type='Int' location=<source>:1:9 range=[<source>:1:9 - line:1:9] nothrow arg_labels=_builtinIntegerLiteral:
(constructor_ref_call_expr implicit type='(_MaxBuiltinIntegerType) -> Int' location=<source>:1:9 range=[<source>:1:9 - line:1:9] nothrow
(declref_expr implicit type='(Int.Type) -> (_MaxBuiltinIntegerType) -> Int' location=<source>:1:9 range=[<source>:1:9 - line:1:9] decl=Swift.(file).Int.init(_builtinIntegerLiteral:) function_ref=single)
(type_expr implicit type='Int.Type' location=<source>:1:9 range=[<source>:1:9 - line:1:9] typerepr='Int'))
(tuple_expr implicit type='(_builtinIntegerLiteral: Int2048)' location=<source>:1:9 range=[<source>:1:9 - line:1:9] names=_builtinIntegerLiteral
(integer_literal_expr type='Int2048' location=<source>:1:9 range=[<source>:1:9 - line:1:9] value=16))))))
(var_decl "x" type='Int' interface type='Int' access=internal let storage_kind=stored))
Woah! There's a lot happening here! Hang in there with me, lets try to decipher this a little bit. Lets start simple:
(var_decl "x" type='Int' interface type='Int' access=internal let storage_kind=stored))
We can see our variable declaration for x still and we notice that there is now a type, Int, and we see the access control level for it as well, internal. That wasn't too bad...
Now for the good stuff!
(pattern_binding_decl
(pattern_named type='Int' 'x')
(call_expr implicit type='Int' location=<source>:1:9 range=[<source>:1:9 - line:1:9] nothrow arg_labels=_builtinIntegerLiteral:
(constructor_ref_call_expr implicit type='(_MaxBuiltinIntegerType) -> Int' location=<source>:1:9 range=[<source>:1:9 - line:1:9] nothrow
(declref_expr implicit type='(Int.Type) -> (_MaxBuiltinIntegerType) -> Int' location=<source>:1:9 range=[<source>:1:9 - line:1:9] decl=Swift.(file).Int.init(_builtinIntegerLiteral:) function_ref=single)
(type_expr implicit type='Int.Type' location=<source>:1:9 range=[<source>:1:9 - line:1:9] typerepr='Int'))
(tuple_expr implicit type='(_builtinIntegerLiteral: Int2048)' location=<source>:1:9 range=[<source>:1:9 - line:1:9] names=_builtinIntegerLiteral
(integer_literal_expr type='Int2048' location=<source>:1:9 range=[<source>:1:9 - line:1:9] value=16))))))
We see that we our simple integer_literal_expr from parsing has now transformed into what appears to be a call_expr. I'll help you out a bit and spell this out in Swift and hopefully you can look at this and kind of understand the structure.
internal let x: Int = Int(_builtinIntegerLiteral: 16)
This is where we see the tight relationship between the compiler and the standard library. You can actually find this initializer for Int here: https://github.com/apple/swift/blob/master/stdlib/public/core/IntegerTypes.swift.gyb#L1111 (forgive the gyb syntax, but trust me when I say this does what it should 
)
This is a trivial example about what Sema does during this stage, but I encourage you to check it out further as this is my favorite step in the pipeline!
An example error during this stage:
// error: cannot convert value of type 'Bool' to specified type 'Int'
let x: Int = true
^~~~
We assigned a Bool value to a variable with the Int type and thankfully the type checker caught this for us!
Sema - https://github.com/apple/swift/tree/master/lib/Sema
Semantic Analysis - https://en.wikipedia.org/wiki/Semantic_analysis_(compilers) (This wiki isnt too helpful, but it includes some useful links to things like type checker)
If you've made it this far, you're in for a wild ride next! SIL provides things like high level optimizations for our Swift code, emitting dataflow diagnostics, and a many more.
// Type checked AST from Sema
internal let x: Int = 16
// SIL
sil_stage canonical
import Builtin
import Swift
import SwiftShims
let x: Int
sil_global hidden [let] @$S6output1xSivp : $Int
sil @main : $@convention(c) (Int32, UnsafeMutablePointer<Optional<UnsafeMutablePointer<Int8>>>) -> Int32 {
bb0(%0 : $Int32, %1 : $UnsafeMutablePointer<Optional<UnsafeMutablePointer<Int8>>>):
alloc_global @$S6output1xSivp // id: %2
%3 = global_addr @$S6output1xSivp : $*Int // user: %6
%4 = integer_literal $Builtin.Int64, 16 // user: %5
%5 = struct $Int (%4 : $Builtin.Int64) // user: %6
store %5 to %3 : $*Int // id: %6
%7 = integer_literal $Builtin.Int32, 0 // user: %8
%8 = struct $Int32 (%7 : $Builtin.Int32) // user: %9
return %8 : $Int32 // id: %9
} // end sil function 'main'
sil public_external [transparent] [serialized] @$SSi22_builtinIntegerLiteralSiBi2048__tcfC : $@convention(method) (Builtin.Int2048, @thin Int.Type) -> Int {
bb0(%0 : $Builtin.Int2048, %1 : $@thin Int.Type):
%2 = builtin "s_to_s_checked_trunc_Int2048_Int64"(%0 : $Builtin.Int2048) : $(Builtin.Int64, Builtin.Int1) // user: %3
%3 = tuple_extract %2 : $(Builtin.Int64, Builtin.Int1), 0 // user: %4
%4 = struct $Int (%3 : $Builtin.Int64) // user: %5
return %4 : $Int // id: %5
} // end sil function '$SSi22_builtinIntegerLiteralSiBi2048__tcfC'
My oh my there is a lot to cover here. There's a lot of interesting parts that we could go over, but for the sake of having a basic understanding of what is happening here I'll go over just the parts in the main function.
sil @main : $@convention(c) (Int32, UnsafeMutablePointer<Optional<UnsafeMutablePointer<Int8>>>) -> Int32
If this function signature looks similar to anyone, you know exactly what this looks like.
int main(int argc, char **argv)
for the body of the main function:
alloc_global @$S6output1xSivp // id: %2
%3 = global_addr @$S6output1xSivp : $*Int // user: %6
%4 = integer_literal $Builtin.Int64, 16 // user: %5
%5 = struct $Int (%4 : $Builtin.Int64) // user: %6
store %5 to %3 : $*Int // id: %6
%7 = integer_literal $Builtin.Int32, 0 // user: %8
%8 = struct $Int32 (%7 : $Builtin.Int32) // user: %9
return %8 : $Int32 // id: %9
Hopefully I can translate directly what this does for a quick overview. First we allocate a global variable, x, and then we store the address of it in %3. We then construct an Int with the value of 16 which is stored in %5. After we construct the integer, we store the value of it to the address of the global variable x. Then the last three lines are simply return 0 for the main function.
It's important to note that there are two steps in this SIL step, SILGen and SIL. SILGen takes the AST from Sema and produces a raw version of SIL that is handed off to SIL for mandatory optimizations and diagnostics which then produces a canonical version of the SIL.
We also get new flags we can pass here as well!
// This emits the raw version of SIL from SILGen
$ swiftc -emit-silgen example.swift
// This emits the canonical version of SIL
$ swiftc -emit-sil example.swift
An example error at this stage:
// error: missing return in a function expected to return 'Int'
func add(_ x: Int, _ y: Int) -> Int {
let sum = x + y
}
^
This is part of the dataflow diagnostics that I mentioned earlier. SIL determines that there is no "terminator", or return instruction in this case, for this function and emits an error letting the developer know.
SIL Header files - https://github.com/apple/swift/tree/master/include/swift/SIL
SIL Diagnostics - https://github.com/apple/swift/blob/master/include/swift/AST/DiagnosticsSIL.def
SIL Optimizer Header files - https://github.com/apple/swift/tree/master/include/swift/SILOptimizer
SILGen Implementation files - https://github.com/apple/swift/tree/master/lib/SILGen
SIL Implementation files - https://github.com/apple/swift/tree/master/lib/SIL
SIL Optimizer Implementation files - https://github.com/apple/swift/tree/master/lib/SILOptimizer
SIL Documentation - https://github.com/apple/swift/blob/master/docs/SIL.rst (I highly recommend reading this as it will do a far better job explaining SIL and its purpose)
SIL Programmer's Manual (Work in Progress) - https://github.com/apple/swift/blob/master/docs/SILProgrammersManual.md
Video with Joe Groff and Chris Lattner at LLVM conf - https://www.youtube.com/watch?v=Ntj8ab-5cvE
I promise we're almost done! This is the final step (at least for the Swift compiler!) in the compilation process. The goal of this step is to take our refined and optimized SIL and emit LLVM IR (LLVM Intermediate Representation).
I won't copy the SIL from the last step here, but I will show you the IR produced from this stage (not all of the IR, but the relevant stuff).
%TSi = type <{ i64 }>
define protected i32 @main(i32, i8**) #0 !dbg !25 {
%2 = bitcast i8** %1 to i8*
store i64 16, i64* getelementptr inbounds (%TSi, %TSi* @"$S6output1xSivp", i32 0, i32 0), align 8, !dbg !30
ret i32 0, !dbg !30
}
This isn't too bad! First we define %TSi as a type containing i64, which is a 64 bit integer in LLVM. I'll spell out what exactly what %TSi is in Swift for a better understanding:
public struct Int {
var _value: Builtin.Int
}
%TSi is just our Int type in Swift!
Next, we define our main method and store i64 16 into our global variable x and finally return i32 0 (return 0).
And, as always we get new flags to emit llvm ir for us:
$ swiftc -emit-ir example.swift
From here, we pass this IR to LLVM to emit a native binary for us. If you want to see the assembly produced from LLVM:
$ swiftc -emit-assembly example.swift
IRGen - https://github.com/apple/swift/tree/master/lib/IRGen
IRGen Diagnostics - https://github.com/apple/swift/blob/master/include/swift/AST/DiagnosticsIRGen.def
Yes, you will need to know some C++ to get started with contributing. Don't panic! The C++ in the compiler isn't overly complex just to be complex, it's actually pretty down to Earth.
This is where my ways of learning how to contribute can be different for folks.
For me, I like to implement a lot of cool features and the best way for me is to look at some of the errors that prevent these features and find them in the diagnostic files. Once I find the specific diagnostic, I find the name of it and look for it in the project. It might be in the parser or Sema where it emits this error and I start from there. I look at the code around it and try to make sense of why its producing an error. This is how I learned most of my C++ by just looking around at the compiler. This method is pretty effective for people because you see an error that bothers you sometimes, and look at the code behind it. You have a relationship (I couldn't find the exact wording for this at the moment) with the error and you can relate to the code that produces it. By learning the compiler this way, you'll notice a lot of helpful comments around the code explaining why its making an error or what in the world its doing. (There are a lot of places without comments, so this is also a fantastic contribution that you can make for future contributors).
For others, it may require you to study C++ for a while to feel comfortable looking at compiler source code. This is totally fine! There are plenty of great material online for you to learn C++.
I almost forgot to mention, the Swift compiler C++ is very closely tied to LLVM C++ which is a lot easier to work in. LLVM C++ isn't an actual language, but its the APIs that help make it more down to Earth.
There are plenty of Starter bugs on the Swift bugs website (https://bugs.swift.org/browse/SR-9218?jql=labels%20%3D%20StarterBug) where you can give squashing bugs a shot.
Harlan Haskins and Robert Widmann gave an excellent presentation about this very topic here - https://www.youtube.com/watch?v=oGJKsp-pZPk
Of course I just gave a brief overview on the whole compilation process, but I hope you learn the various places where errors or bugs can occur. It's also very helpful when you want to implement a new feature because now you hopefully have a rough idea where to start (For example, if you want a new automatic derived conformance, you can look in Sema which has a lot of derived conformance files for you to look at). The Swift Readme can guide you through building the project and getting a project structure ready for you to start contributing effectively.
Hopefully this wasn't too much at once, but I look forward to everyone contributing to Swift!
It seems that people already gave you some amazing pieces of advice, and although I'm no compiler expert I relate to you and wanted to answer this in my perspective as I was pretty much in the same position a few months ago. I have an iOS career, never touched compilers, knew C++ from college but never really did much with it, and sure as hell wanted to contribute to Swift.
I opened my first PR in march, and although I can't say I became an expert in compilers, this experience taught me a lot and I'm now happy to be able to somewhat follow the discussions regarding new features (in terms of necessary compiler development). I find this amazing and I'm now working towards learning more about compilers as well so I can work on more difficult tasks.
I went into this journey completely in the dark (I just had a very shallow idea of how the Swift compiler worked in terms of steps), and what I did was focus on something very specific that I wanted to fix. In my case, that ended up being SourceKit because I wanted to fix Xcode's highlighting issues. I had a list of cases where Xcode failed to do that, opened bugs in the JIRA, tried to determine which one of them was easier to fix and focused on it.
You'll see that the compiler is enourmous, but you can attach lldb to it and its tools. What I did was search the repo for occurrences of things that could be relevant to my problem ("highlight", "index", SourceKit's keys), created a bunch of breakpoints and started scanning the backtraces until I managed to find pieces of code that seemed to have something to do with my problem. It was a painful process in the beginning as I really had no idea how the compiler worked, but the docs available on the repo were very useful when I was stuck.
In the end, I always ended up finding what I needed to fix. I eventually ended up gaining more knowledge on compilers and the whole process got a lot easier, but it's how I worked on my first few bug fixes. Today, I got all of those bugs from my list fixed.
Although my process was noobish at best, I thought you would find it interesting considering I was at the same position as you. I had this feeling that the compiler was a pro-only zone and everyone would laugh at me if I tried to help, but it's like all other types of software. The more you look at it the easier it gets.
The veterans will give you tons of information about the compiler, but the most helpful information I got during this journey is that you can always ask for help. Hell, I've even sent random tweets to the core team asking for random Swift stuff and always got extremely helpful answers. The Swift team is probably the most "open" team at Apple, and the whole community is very helpful.
Although the docs folder doesn't contain an overall detailed picture, there are some excellent documents on various parts of the compiler (for example Testing.md, TypeChecker.rst and Diagnostics.md), and the compiler source is moderately documented as well. It is best to read them step by step as you gain experience and target new components in practice rather than going for the whole thing at once. Not too long ago, I found the advice to start with Lexicon.rst very helpful. The good news is that simple tasks do not require you to understand the details of what happens beyond the component you are working in.
If you're experienced in Swift and have some intermediate c++ skills, my advice is to pick a starter bug and just make that one thing work © (thanks, @jrose). Get the compiler building, write a test that reproduces the bug, feed it to the compiler and start groping for what could make it work. As @codafi says, swiftc is not an accurate embodiment of a formal compilers class. Simply treat it as any other large app if you're a developer, and naturally, feel free to ask questions and reach out for people from the Swift team.
You bring up some good resources, but I want to caution that these docs are really young - I have voiced my concerns about TypeChecker.rst in particular being inaccurate before but I won't rehash that here.
Thats a fantastic overview, thanks for taking the time to write it.
I recently made a small contribution to the standard library which involved me using Xcode (as that's what I'm used to) and VS Code for the tests, then running the command line build utility. That definitely didn't seem like the optimum setup for doing what I was doing but it worked for me at the time. What is the typical setup for people who work on the swift compiler projects on a day to day basis?
So cool, that is question I wanna ask so far. And I wonder what would be development environment usually used for swift compiler. e.g. which operating system(linux or macOS), source code editor(emacs or vim). I am new to swift compiler as well, and hope in near future I can make some contribution to swift compiler as well.
Thank you all so much for your help 
I thought I'd be walking away from this with a stack of textbooks and a Udacity course or two, but I'm walking away with a ton of optimism instead!
Based on the advice of @codafi, @rockbruno, and @Alejandro, my current plan is to just pick a bug and start picking away at it. Just like you mentioned @rockbruno, one of my motivations going into this is to be able to fix bugs that I come across in my own Swift usage, so I figure it makes sense to pick one of those to start.
I'm looking at [SR-9217] Incorrect error message when the result of a keypath doesn't match the return type of a function · Issue #51706 · apple/swift · GitHub right now because the reproduction is nice and short and I've got an error message to work with which seems like a plus based on what @Alejandro said.
I'm going to be following @harlanhaskins and @codafi's talk to get my machine set up with the compiler. I don't want to speak too soon, but I just watched the whole thing, and it seemed like it was super clear and well done.
I'm definitely going to be reaching out for help and advice (I see your DM @codafi).
Seriously, thank you so much everyone!
This thread is a fantastic overview!
I have a similar style of learning to what @codafi mentioned, getting in and doing something. Although I’ve found the biggest barrier there is the long build and iteration times.
I think many would find it super useful to know how to discover the best build incantation for each part of the compiler, or other tips and tricks for a good development loop. Possibly broken down by the part you’re working on like @Alejandro’s excellent overview. Does such a resource exist?
Thank you @codafi and @Alejandro, it's really great to see that while compilers are dense they aren't completely unapproachable, as @twof says it's really great to have such encouraging responses to our questions.
@Alejandro Your overview is so great! It will be really useful whenever I need to get back the general picture of Swift compile pipeline in my head! 
I started working on this bug and thanks to the advices of @codafi I was able to hack things and to actually make it (nearly) work, I have been progressing in solving this issue and that's really encouraging.
Thank you @anthonylatsis for those links, this will come in handy in the upcoming work I'll do!