Extending functionality of Build Server Protocol with SourceKit-LSP

Development SourceKit-LSP
1

Extending functionality of Build Server Protocol with SourceKit-LSP

SourceKit-LSP has been able to connect to a build server through the Build Server Protocol (BSP) to get build settings for a source file and thus provide intelligent editor functionality for a few years. Recently, SourceKit-LSP has gained a deeper understanding of a project’s structure (in particular targets and target dependencies) to support background indexing in SwiftPM projects. We would like to open this capability up to other build systems by extending SourceKit-LSP’s interaction through BSP, building on top of SourceKit-LSP for other build systems - #12 by blangmuir with new knowledge that we gained by implementing background indexing for SwiftPM projects.

The goal is that BSP is used for all interactions between SourceKit-LSP and build systems, including the built-in build systems (SwiftPM and compile_commands.json) so that all functionality is guaranteed to be exposed to external build systems.

As we implement the integration with BSP and learn more about the interaction, the exact details might change and this post might become outdated. We plan to have some reference documentation in SourceKit-LSP that describes the necessary requests that need to be implemented by a BSP server for SourceKit-LSP, which should be kept up-to-date.

Current BSP Integration

For background, the current interaction between SourceKit-LSP and a BSP server works as follows:

  • SourceKit-LSP and BSP create the connection using the BSP build/initialize request. The BSP server returns the path to the index store and IndexStoreDB as part of that request, so that SourceKit-LSP can provide index-based cross-module functionality
  • When a source file is opened, SourceKit-LSP sends the custom textDocument/registerForChanges request to the BSP server, which notifies the build server that it is interested in build settings for that file. When the document is closed, it sends a textDocument/registerForChanges request again, with the unregister action.
  • When the build server has build settings for a source file, it sends a custom build/sourceKitOptionsChanged notification to SourceKit-LSP, pushing the new build settings to SourceKit-LSP.

The only standard BSP request that is currently in use build/initialize.

Proposed new BSP Integration

BSP already has requests that can represent the structure of targets in the project and we are planning to re-use those for SourceKit-LSP. BSP servers will still need to implement some custom requests to support SourceKit-LSP, such as a request to get build settings. The following is split into 4 sections:

  • The standard BSP request that we will start using. Existing BSP servers likely already implement these. BSP servers that were created to only serve command line options to SourceKit-LSP would need to adopt these.
  • BSP extensions that are required to be implemented by the BSP server so that SourceKit-LSP can function properly.
  • BSP extensions that are needed to support background indexing. SourceKit-LSP will function using index-while-building without these requests.
  • Additional requests that allow the BSP server to further customize behavior but that aren’t required.

Standard BSP requests

  • build/initialize Continue to use the standard request to initialize the connection like we are today, including sending indexStorePath and indexDatabasePath in the data field from the BSP server to SourceKit-LSP, like we do today.
  • buildTarget/inverseSources: Request from SourceKit-LSP to BSP to get the targets a file belongs to.
    • Caching: The result of this request is assumed to be stable and cashable until the BSP server sends any buildTarget/didChange notification to SourceKit-LSP. Every buildTarget/didChange notification needs to invalidate the cache since the changed target might gained a source file.
  • buildTarget/sources: Request from SourceKit-LSP to the BSP server to get the sources in a target, including the following optional extension:
    • BSP servers can set the optional language?: LanguageId field in the SourcesItem to specify the language that should be used for background functionality. Otherwise, the language is inferred from the file extension.
    • Caching: The result of this request is assumed to be stable and cashable until the BSP server sends a buildTarget/didChange to SourceKit-LSP for this target.
  • buildTarget/didChange: Notification from BSP to SourceKit-LSP to notify that a build target has changed. This causes the target’s file list and the build settings for open files in the target to be reloaded. As an extension, we allow changes to be null to indicate that all build targets have changed. This is useful for build systems that don’t have target-level tracking of changes, eg. after a Package.swift was reloaded in SwiftPM.
  • workspace/buildTargets: Request to get all targets in the project.
    • BSP servers should set the "test" tag for targets that might contain tests so these targets are included in the test navigator.
    • BSP servers can add the custom "dependency" tag to a BuildTarget to exclude tests from a target from the test navigator. This can eg. be used for test targets of SwiftPM package dependencies.
    • Caching: The result of this request is assumed to be stable and cacheable until the BSP server sends a buildTarget/didChange to SourceKit-LSP for any target.
  • window/logMessage: Notification from the BSP server to SourceKit-LSP that allows logging a message to the editor’s log. SourceKit-LSP forwards this to the editor.

Note that build systems that don’t have a notion of targets (eg. you can consider compile_commands.json as such a build system), it is acceptable that they only provide a single dummy target for all files.

BSP Extensions required for SourceKit-LSP functionality

textDocument/sourceKitOptions

/** Request from SourceKit-LSP to the BSP server to retrieve the compiler 
  * arguments that should be used to build the given file in the given target */
export interface BuildSettingsParams {
  textDocument: TextDocumentIdentifier;
  target: BuildTargetIdentifier;
}

export interface BuildSettingsResponse {
  /** The arguments that should be passed to a compiler to build this file */
  compilerArguments: string[];
  
  /** The working directory in which the compiler should be run. 
   * If `null`, the file should be built without a working directory. */
  workingDirectory?: string;
}

This is the key request that BSP servers need to implement for SourceKit-LSP. It provides the compiler arguments that are used to provide semantic functionality for the file. This was originally proposed in SourceKit-LSP for other build systems - #12 by blangmuir but never implemented.

Closest BSP equivalent: buildTarget/cppOptions in the C++ extension of BSP but its structure is quite different because it eg. explicitly specifies defines aka. -D flags. textDocument/sourceKitOptions is more opaque.

Caching: The result of this request is assumed to be stable and cacheable until the BSP server sends a buildTarget/didChange for the target passed in this request.

As described above, the current BSP integration pushes build settings from the build server to the client. SourceKit-LSP has been redesigned to be more pull-based. This pull-based model fits better to recent developments in LSP where eg. diagnostics also became a pull request, initiated by the editor, instead of the LSP server pushing diagnostics. We would thus like to deprecate the existing build/sourceKitOptionsChanged notification from the BSP server to the client in favor of this pull model.

BSP Extensions required for Background Indexing

buildTarget/prepare

/** Build Swift modules for this target so that source files in downstream 
 * modules can import modules defined by this target */
export interface PrepareParams {
  /** A sequence of build targets to compile. */
  targets: BuildTargetIdentifier[];
  /** A unique identifier generated by the client to identify this request.
   * The server may include this id in triggered notifications or responses. */
  originId?: OriginId;
}

export interface PrepareResult {}

The closest BSP equivalent is buildTarget/compile but we have decided to use a different method because preparation doesn’t need to generate any binaries and the generated Swift modules only need to contain declarations, no bodies. It is thus different from a compile.

In the future, we could consider running buildTarget/compile if the build server doesn’t support buildTarget/prepare in order to try preparing a module. The problem is that a normal compile usually doesn’t continue building if a target’s dependency fails to build but preparation is expected to continue here.

workspace/waitForBuildSystemUpdates

This is a no-op. If the build system is currently processing updates, like file changes or is currently building the build graph, it should only return once they have finished processing. This is used by background indexing for eg. the current scenario:

  • A new source file is added to the project and should thus be indexed in the background.
  • We notify the build system about the new file.
  • We want to wait until the build system has incorporated the file into its build system so it can provide build settings for it.

BSP Extensions to help BSP Servers

workspace/didChangeWatchedFiles

Correctly watching for file changes on all platforms is non-trivial. To help BSP servers that are developed with SourceKit-LSP as its primary user, SourceKit-LSP can forward the LSP notification workspace/didChangeWatchedFiles from the editor to the BSP server. To receive the file change notifications, the BSP server must set the watches: FileSystemWatcher[] parameter in the initialize response, specifying the glob patterns to watch for.

9 Likes
2

Sounds interesting. I'm not familiar with BSP, but I certainly interested in the possibility of SourceKit-LSP supporting alternative build systems besides SwiftPM.

3

This is great! I have been recently looking into expanding sourcekit-lsp to work with Bazel projects, and it seems this is exactly what I would need in this case.

4

very happy to hear about ongoing work on SourceKit-LSP. thank you!

i use neovim as my daily driver, will these changes improve the problem of sourcekit not understanding Xcode projects? i currently architect all my projects to have almost all my code in SPM packages and only thin entrypoints calling out to SPM modules within Xcode projects, because that way i can use neovim with the LSP.

5

Extending the Build Server Protocol with SourceKit-LSP involves integrating SourceKit-LSP's language features into the build process to enhance development workflows. This integration allows for improved code analysis, diagnostics, and tooling support, offering a more cohesive experience for developers working with Swift and other languages supported by SourceKit.