Getting Started with Swift Compiler Development

Development Compiler
2

I have tried to setup the compiler before on my computer, but due to my limited command line knowledge, I have had trouble doing it properly as I fundamentally don't understand some of the commands I am or should be running in the compiler installation process. Also, I'm not totally sure where the compiler needs to be located on my computer. Any help in these areas would be greatly appreciated!

[The Readme has the exact steps with flags but here is the outline.]

  1. Install cmake and ninja.
  2. Clone and run update-checkout.
  3. Run swift/utils/build-script --release-debuginfo --xcode. [Add a --debug-swift to get better debugging info for the compiler itself. Other dependencies like LLVM will have debug info but will have more optimizations applied with --release-debuginfo.]. This will create an xcodeproj under build/Xcode-RelWithDebInfoAssert/swift-macosx-x86_64.
  4. Open the Xcode project and under schemes, do "Manually manage schemes" and select ALL_BUILD as well as swift (this list is very long so you'll have to scroll a bit).
  5. Build normal using Cmd+B inside Xcode to double-check that it works.
  6. Check out a new branch and make your changes.
  7. Run tests using lit on the commandline. I recommend using something like
    llvm-project/llvm/utils/lit/lit.py -s -vv build/Xcode-RelWithDebInfoAssert/swift-macosx-x86_64/test-macosx-x86_64 --filter "myTestName". There are some test targets which you can run from inside Xcode, but I usually use lit because you can change the flags. I highly recommend using -vv, regardless of what you do, so that you get to see which line in a multi-line test failed.
  8. Your test passes yay! Now remove the --filter and make sure you didn't break anything else.
  9. Submit a PR and @ someone, probably someone who is helping you make the change.

Upon somehow downloading the compiler previously, I was extremely overwhelmed and had no idea where to start. The compiler just has so many files and folders and I am really unsure of exactly what each part does.

You could start off by working on a bug labeled StarterBug on JIRA. https://bugs.swift.org . Usually, we try to provide a bunch of instructions to help you get started but (regardless) you should definitely ask lots of questions you go -- the codebase is large and wrapping your head around it will take time. You might have questions like:

  1. Where should I add this test?
  2. How should I write this test?
  3. What kind of tests should I write?
  4. Where should I look for doing X?
  5. Should this be a new file or its own file?
  6. How do I change the CMake to make it pick up my new stuff?
  7. This test is failing and I don't know why? Help!
  8. Can you please re-run the CI for me?
  9. Could you provide more detailed steps for how to do X?
  10. How do I use lit to do what I want?
  11. How do I use FileCheck or the other things I keep seeing in test files?
  12. Where can I find documentation on X?
  13. I'm having trouble understanding the CI failure, can you help?
  14. I don't understand why the CI fails but it passes locally for me?

and so on. All of these are very reasonable and we understand that contributing to a large project is hard. Please feel free to re-ping people (or post here) in case you felt like your question slipped through.

For compiler-internal jargon, docs/Lexicon.rst should be helpful. If you're hacking around the AST and SIL, check out the corresponding documentation and these two blog posts: The secret life of types in Swift | by Slava Pestov | Medium
How to talk to your kids about SIL type use | by Slava Pestov | Medium

At the same time, please feel free to submit feedback as you go (say, "I tried to use update-checkout like so but it failed with X error which seems unclear. Could we change the message to Y instead?").

Furthermore, I am unfamiliar with the most of the underlying workings of Swift; by that I mean like, for example, how protocols conformances are tracked, how syntax is parsed, and really how most of the internal details like these work.

I've been working on the compiler full-time for the past ~5 months and I could at best give you a vague answer to both of those questions. I think it would be more helpful to focus on answering small, specific questions that help you fix one small issue at a time, without trying to understand how the whole compiler works in one go.

Also I am not a very experienced debugger and don't know how to really trace my way back to an error (using REPL I think?) to a meaningful level.

Personally, I usually use a mixture of print debugging + the Xcode debugger and build the compiler with something like swift/utils/build-script --release-debuginfo --xcode --debug-swift. This means that compile times for the standard library are longer, but all the debug info is present which makes things much easier for me. Others don't use --debug-swift and use lldb from the commandline. build-script has lots of other flags which allow more granularity.

I am willing to put in the necessary time and energy in to learning how all of this stuff works so I can hopefully become somewhat versed in compiler development and play a bigger role in the evolution of Swift

Thank you and we look forward to seeing you more often here and on JIRA and GitHub. :smile:

Other tips that I can think of the top off my head:

  1. Sometimes you'll see build failures which seem utterly inexplicable/unclear -- you should probably ask someone for help but some steps you can take in the meanwhile (increasingly nuclear): (1) use the same build-script invocation as earlier but use the --reconfigure flag, (2) delete the CMakeLists.txt (find . -iname 'CMakeCache.txt' -type f -delete) from swift-macosx-x86_64 and re-run build-script (3) delete the entirely build directory.
  2. If you're rebuilding a lot, consider using sccache. See docs/DevelopmentTips.md for details.
  3. If you're seeing a CI failure, especially after force-pushing, where it is like "the tests failed, but I'm restarting the tests", it is probably not your fault so don't worry about it. Same goes if you see a Java exception.
  4. I keep a journal to keep track of context. That way, if I stop working on something for a few days (or weeks) and get back to it later, I can read it and get up to speed quicker.
  5. Tell us when you start working on something! That helps avoid duplicated effort.

All that said and done, I think there's one key thing I want to point out, which usually isn't specified in a Readme. If you start working on a task but feel that you will not have the time for it, you should feel zero shame or embarrassment in telling us so. That is perfectly fine. Life happens. There are many things in life that are more important than a compiler.

12 Likes