Getting Started with Swift Compiler Development

To start off, I have been an avid user of Swift for the past 2 years. I am very familiar with the Swift language and want to become more involved in its evolution. I would like to be able to contribute to discussions on a more meaningful level and be able to propose feature backed by an implementation. In pursuit of this, I have run into the inevitability that I will need to work with and understand the Swift compiler.

I have limited experience in the command line itself and can for the most part just do basic things like use Homebrew, git, cocoa pods, and basic file navigation and manipulation. Also, I have limited knowledge of C++ and Python and can only really right basic programs in the two. I am willing and motivated to learn more in all of these areas and I assume I will probably become more knowledgeable in all of them through compiler development.

Here is some information about my current setup:

  • I am working off of a 13-inch, 2019 Macbook Pro with a touch tar
  • I am currently running macOS Catalina, version 10.15.1
  • My shell is zsh

I have tried to setup the compiler before on my computer, but due to my limited command line knowledge, I have had trouble doing it properly as I fundamentally don't understand some of the commands I am or should be running in the compiler installation process. Also, I'm not totally sure where the compiler needs to be located on my computer. Any help in these areas would be greatly appreciated!

On a previous computer though, I did manage to download the compiler, after which I ran into a few barriers. For one, I wasn't really sure how to develop the compiler. By that I mean like does it open in some massive Xcode project? If so, how do I run my version of the compiler to test it? Also, just a stylistic question, but where should I store my test swift files (playground or command line project I assume?) on my computer?

Upon somehow downloading the compiler previously, I was extremely overwhelmed and had no idea where to start. The compiler just has so many files and folders and I am really unsure of exactly what each part does. Furthermore, I am unfamiliar with the most of the underlying workings of Swift; by that I mean like, for example, how protocols conformances are tracked, how syntax is parsed, and really how most of the internal details like these work. Also I am not a very experienced debugger and don't know how to really trace my way back to an error (using REPL I think?) to a meaningful level.

I am willing to put in the necessary time and energy in to learning how all of this stuff works so I can hopefully become somewhat versed in compiler development and play a bigger role in the evolution of Swift, but where I stand, I am overwhelmed and feel under-informed about the compiler despite the fact that I really want to get into it.

Any advice, direction, or instruction for a beginner like me?

Any and all help is appreciated! Thanks!

6 Likes

I have tried to setup the compiler before on my computer, but due to my limited command line knowledge, I have had trouble doing it properly as I fundamentally don't understand some of the commands I am or should be running in the compiler installation process. Also, I'm not totally sure where the compiler needs to be located on my computer. Any help in these areas would be greatly appreciated!

[The Readme has the exact steps with flags but here is the outline.]

  1. Install cmake and ninja.
  2. Clone and run update-checkout.
  3. Run swift/utils/build-script --release-debuginfo --xcode. [Add a --debug-swift to get better debugging info for the compiler itself. Other dependencies like LLVM will have debug info but will have more optimizations applied with --release-debuginfo.]. This will create an xcodeproj under build/Xcode-RelWithDebInfoAssert/swift-macosx-x86_64.
  4. Open the Xcode project and under schemes, do "Manually manage schemes" and select ALL_BUILD as well as swift (this list is very long so you'll have to scroll a bit).
  5. Build normal using Cmd+B inside Xcode to double-check that it works.
  6. Check out a new branch and make your changes.
  7. Run tests using lit on the commandline. I recommend using something like
    llvm-project/llvm/utils/lit/lit.py -s -vv build/Xcode-RelWithDebInfoAssert/swift-macosx-x86_64/test-macosx-x86_64 --filter "myTestName". There are some test targets which you can run from inside Xcode, but I usually use lit because you can change the flags. I highly recommend using -vv, regardless of what you do, so that you get to see which line in a multi-line test failed.
  8. Your test passes yay! Now remove the --filter and make sure you didn't break anything else.
  9. Submit a PR and @ someone, probably someone who is helping you make the change.

Upon somehow downloading the compiler previously, I was extremely overwhelmed and had no idea where to start. The compiler just has so many files and folders and I am really unsure of exactly what each part does.

You could start off by working on a bug labeled StarterBug on JIRA. https://bugs.swift.org . Usually, we try to provide a bunch of instructions to help you get started but (regardless) you should definitely ask lots of questions you go -- the codebase is large and wrapping your head around it will take time. You might have questions like:

  1. Where should I add this test?
  2. How should I write this test?
  3. What kind of tests should I write?
  4. Where should I look for doing X?
  5. Should this be a new file or its own file?
  6. How do I change the CMake to make it pick up my new stuff?
  7. This test is failing and I don't know why? Help!
  8. Can you please re-run the CI for me?
  9. Could you provide more detailed steps for how to do X?
  10. How do I use lit to do what I want?
  11. How do I use FileCheck or the other things I keep seeing in test files?
  12. Where can I find documentation on X?
  13. I'm having trouble understanding the CI failure, can you help?
  14. I don't understand why the CI fails but it passes locally for me?

and so on. All of these are very reasonable and we understand that contributing to a large project is hard. Please feel free to re-ping people (or post here) in case you felt like your question slipped through.

For compiler-internal jargon, docs/Lexicon.rst should be helpful. If you're hacking around the AST and SIL, check out the corresponding documentation and these two blog posts: The secret life of types in Swift | by Slava Pestov | Medium
How to talk to your kids about SIL type use | by Slava Pestov | Medium

At the same time, please feel free to submit feedback as you go (say, "I tried to use update-checkout like so but it failed with X error which seems unclear. Could we change the message to Y instead?").

Furthermore, I am unfamiliar with the most of the underlying workings of Swift; by that I mean like, for example, how protocols conformances are tracked, how syntax is parsed, and really how most of the internal details like these work.

I've been working on the compiler full-time for the past ~5 months and I could at best give you a vague answer to both of those questions. I think it would be more helpful to focus on answering small, specific questions that help you fix one small issue at a time, without trying to understand how the whole compiler works in one go.

Also I am not a very experienced debugger and don't know how to really trace my way back to an error (using REPL I think?) to a meaningful level.

Personally, I usually use a mixture of print debugging + the Xcode debugger and build the compiler with something like swift/utils/build-script --release-debuginfo --xcode --debug-swift. This means that compile times for the standard library are longer, but all the debug info is present which makes things much easier for me. Others don't use --debug-swift and use lldb from the commandline. build-script has lots of other flags which allow more granularity.

I am willing to put in the necessary time and energy in to learning how all of this stuff works so I can hopefully become somewhat versed in compiler development and play a bigger role in the evolution of Swift

Thank you and we look forward to seeing you more often here and on JIRA and GitHub. :smile:


Other tips that I can think of the top off my head:

  1. Sometimes you'll see build failures which seem utterly inexplicable/unclear -- you should probably ask someone for help but some steps you can take in the meanwhile (increasingly nuclear): (1) use the same build-script invocation as earlier but use the --reconfigure flag, (2) delete the CMakeLists.txt (find . -iname 'CMakeCache.txt' -type f -delete) from swift-macosx-x86_64 and re-run build-script (3) delete the entirely build directory.
  2. If you're rebuilding a lot, consider using sccache. See docs/DevelopmentTips.md for details.
  3. If you're seeing a CI failure, especially after force-pushing, where it is like "the tests failed, but I'm restarting the tests", it is probably not your fault so don't worry about it. Same goes if you see a Java exception.
  4. I keep a journal to keep track of context. That way, if I stop working on something for a few days (or weeks) and get back to it later, I can read it and get up to speed quicker.
  5. Tell us when you start working on something! That helps avoid duplicated effort.

All that said and done, I think there's one key thing I want to point out, which usually isn't specified in a Readme. If you start working on a task but feel that you will not have the time for it, you should feel zero shame or embarrassment in telling us so. That is perfectly fine. Life happens. There are many things in life that are more important than a compiler.

12 Likes

I really like this idea. Im going to have to add this to my workflow from now on.

1 Like