How to use SourceKit-LSP to develop the Swift compiler repo?

Development SourceKit-LSP
, ,
1

I have the Swift compiler repo built with Ninja and tried to use SourceKit-LSP on Visual Studio Code to develop the Swift compiler repo. The clangd arguments forwarding works so far so good. But the Swift files can only get compiled with fallback arguments which makes the definition jumping to be constrained in a single file:

[2023-07-15 10:00:22.830] {
  key.request: source.request.cursorinfo,
  key.compilerargs: [
    "-sdk",
    "/Applications/Xcode-15.0-beta-2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.0.sdk",
    "/Users/REDACTED/swift-project/swift/stdlib/public/core/Dictionary.swift"
  ],
  key.offset: 21037,
  key.sourcefile: "/Users/REDACTED/swift-project/swift/stdlib/public/core/Dictionary.swift",
  key.retrieve_refactor_actions: 1
}

This is my arguments for SourceKit LSP:

  "sourcekit-lsp.serverArguments": [
    "--log-level",
    "debug",
    // "--scratch-path",
    // "../build/Ninja-DebugAssert/swift-macosx-arm64",
    // clangd began
    "-Xclangd",
    "--compile-commands-dir=../build/Ninja-DebugAssert/swift-macosx-arm64/",
    "-Xclangd",
    "--query-driver=/Applications/Xcode-15.0-beta-2.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang",
    "-Xclangd",
    "--all-scopes-completion",
    "-Xclangd",
    "--cross-file-rename",
    "-Xclangd",
    "--pch-storage=memory",
    "-Xclangd",
    "--background-index",
    "-Xclangd",
    "-j=10",
    "-Xclangd",
    "--inlay-hints=true",
    // clangd ended
  ],

I also have made an alias of compile_commands.json in Swift's build directory root to the root of the Swift compiler's project root. SourceKit LSP correctly read the IndexStore in Swift's build directory. But it does not work after all.

It seems that the build commands for particular Swift files in Swift compiler's repo like Dictionary.swift is stored in build.ninja but not compile_commands.json. But there is no such an implemented build server for ninja. I'm guessing that we cannot make SourceKit LSP work with Swift files in Swift compiler's repo before having a build server that implements build server protocol for Ninja. If that's wrong, how can I make SourceKit LSP work with those Swift files.

2

So the short answer is, @compnerd and I haven't hooked up the pieces needed for CMake to generate the compile commands for individual files yet, and the compile command injected in the meantime is the do-nothing : command. Unfortunately, there's not really a great way to get LSP at the moment. For individual files, you can add the commands yourself, but that's pretty painful.

Little bit longer threads to pull on if you're keen. You can get some help from ninja in generating the compilation database. There's an extra : && and && : prepended and appended to the swift commands, but you can sort of use ninja -t compdb > compile_commands to get the listing of the swift invocations that Ninja invokes from the build.ninja file. This is a very raw translation though. You'll notice that only one Swift file per module is actually listed in the generated compilation database, so you won't get completions everywhere. If you're interested in specific files though, you can find the module that it's part of and pull out the command that is used to build it. It's still pretty manual, but at least you don't have to guess about what the command looks like.

Now for the long answer about why this is a bit tricky. So Swift is represented as a linker in the CMake build graph due to the complexity of modeling the Swift incremental builds. So instead of trying to generate individual object files, we (CMake/Ninja) hand swiftc all the Swift files at once and let it figure out what to do to determine what needs to be rebuilt and when. Digressing a bit, it's actually because we treat the swift compiler as a linker that the ninja-generated compilation database has : && and && : surrounding the command. If you set PRE_LINK and POST_LINK commands as a custom command, CMake will replace the : with the commands specified there.

In contrast to Swift generating everything in one step, C and C++ represent the "build" of the object file and the "link" of the objects into static archives, dynamic libraries, and executables separately. Separating the build and link steps makes generating the compile commands-per-file much easier. Talking about the anatomy of the entries, the compilation database generated by CMake is a list of objects containing four elements, the working directory, the command used to generate the object file, the source file itself, and the output generated.

[
...
  {
    "directory": "/my/working/directory",
    "command": "clang++ -I ./relative_to_working_directory/include lib/foo.cpp -o build/lib/foo.cpp.o"
    "file": "lib/foo.cpp",
    "output": "build/lib/foo.cpp.o"
  },
...
]

If a file is part of multiple targets with different flags, there are multiple entries for that file in the compilation database with a command representing how that file is built in each target (for if the different targets have different search directories or something).

So that works great when you have a separate build command for each file, but not so well for Swift. Swift modules are the unit of compilation, and the driver figures out how to schedule them instead of Ninja and the build system, so we don't have a single input file per "compile" in Swift. In fact, at the point where CMake is generating the JSON objects for the compile-commands file, we haven't even considered Swift yet and haven't computed the flags for building the Swift files at all, so when CMake comes asking, the only thing we can reply with is "¯\_(ツ)_/¯, dunno". To keep the other parts working and have a valid database for the bits that are available, we still give it a command, :, which is morally equivalent to calling /usr/bin/true, but it doesn't do anything for LSP in the Swift files. We actually have a similar issue for object libraries (add_library(foo OBJECT bar.swift) doesn't do anything at the moment) since we don't really have a way to generate single object files for the same reason. I haven't had a chance to really dig into it, but we'll need to teach CMake to compute the build command for the whole module, then have a mapping from source file to all of the module that it's a part of, and then grab the command used for each module that the file is a part of and splat that down in the compilation database, and in theory everything should be happy and just work™. I hope that was at least somewhat interesting, even if it didn't actually get you a solution.

3 Likes
3

Thanks for your explanation. The solution to make things just work ™ needs no small effort.

Anyway, I still have a question after read up your reply: the constraint that the compilation unit in Swift is a module but not a single file seemed is caused by Swift source files and their "include" info are no longer sufficient for being a complete compilation unit which is compared to C/C++/Objective-C, since those C-family language source files have explicit include info inside and outside the "module" to make themselves complete. Did I get the point?

4

Yeah, pretty much.

1 Like
5

Since it's getting hearts, here's an update since the post above: CMake 3.29 and newer knows how to generate the compile commands for Swift now.
Verify that either

  • The cmake_minimum_required is 3.29 or newer
  • Or that you have set CMP0157 to NEW when it's available after the cmake_minimum_required call.
    e.g.
cmake_minimum_required(VERSION 3.26)
...
if(POLICY CMP0157)
  cmake_policy(SET CMP0157 NEW)
endif()

Just like with C/C++, CMake will generate a compile_commands.json file that you can symlink into your workspace and SourceKit-LSP should pick it up.

This mode did involve re-writing how CMake models Swift builds, so there are some things to check. Namely that you're using the CMAKE_Swift_COMPILATION_MODE to make a target use whole-module optimizations instead of passing the -wmo flag independently. Otherwise, it should mostly just work.

For more info on improvements for Swift in 3.29: Updates for Swift in the upcoming CMake 3.29

Now just need to update the compiler repo to actually take advantage of the new features. :)

1 Like