# Extending functionality of Build Server Protocol with SourceKit-LSP

**URL:** <https://forums.swift.org/t/extending-functionality-of-build-server-protocol-with-sourcekit-lsp/74400>\
**Category:** SourceKit-LSP\
**Created:** [September 5, 2024, 8:43pm UTC](https://forums.swift.org/t/extending-functionality-of-build-server-protocol-with-sourcekit-lsp/74400 "2024-09-05T20:43:01Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![ahoppen](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/ahoppen/32/36046_2.png) [@ahoppen](https://forums.swift.org/u/ahoppen)\
**Post date:** [September 5, 2024, 8:43pm UTC](https://forums.swift.org/t/extending-functionality-of-build-server-protocol-with-sourcekit-lsp/74400/1 "2024-09-05T20:43:01Z")

</div>

# Extending functionality of Build Server Protocol with SourceKit-LSP

SourceKit-LSP has been able to connect to a build server through the [Build Server Protocol (BSP)](https://build-server-protocol.github.io/) to get build settings for a source file and thus provide intelligent editor functionality for a few years. Recently, SourceKit-LSP has gained a deeper understanding of a project’s structure (in particular targets and target dependencies) to support background indexing in SwiftPM projects. We would like to open this capability up to other build systems by extending SourceKit-LSP’s interaction through BSP, building on top of [SourceKit-LSP for other build systems - #12 by blangmuir](https://forums.swift.org/t/sourcekit-lsp-for-other-build-systems/27109/12) with new knowledge that we gained by implementing background indexing for SwiftPM projects.

The goal is that BSP is used for all interactions between SourceKit-LSP and build systems, including the built-in build systems (SwiftPM and compile\_commands.json) so that all functionality is guaranteed to be exposed to external build systems.

As we implement the integration with BSP and learn more about the interaction, the exact details might change and this post might become outdated. We plan to have some reference documentation in SourceKit-LSP that describes the necessary requests that need to be implemented by a BSP server for SourceKit-LSP, which should be kept up-to-date.

## Current BSP Integration

For background, the current interaction between SourceKit-LSP and a BSP server works as follows:

- SourceKit-LSP and BSP create the connection using the BSP `build/initialize` request. The BSP server returns the path to the index store and IndexStoreDB as part of that request, so that SourceKit-LSP can provide index-based cross-module functionality
- When a source file is opened, SourceKit-LSP sends the custom `textDocument/registerForChanges` request to the BSP server, which notifies the build server that it is interested in build settings for that file. When the document is closed, it sends a `textDocument/registerForChanges` request again, with the `unregister` action.
- When the build server has build settings for a source file, it sends a custom `build/sourceKitOptionsChanged` notification to SourceKit-LSP, pushing the new build settings to SourceKit-LSP.

The only standard BSP request that is currently in use `build/initialize`.

## Proposed new BSP Integration

BSP already has requests that can represent the structure of targets in the project and we are planning to re-use those for SourceKit-LSP. BSP servers will still need to implement some custom requests to support SourceKit-LSP, such as a request to get build settings. The following is split into 4 sections:

- The standard BSP request that we will start using. Existing BSP servers likely already implement these. BSP servers that were created to only serve command line options to SourceKit-LSP would need to adopt these.
- BSP extensions that are required to be implemented by the BSP server so that SourceKit-LSP can function properly.
- BSP extensions that are needed to support background indexing. SourceKit-LSP will function using index-while-building without these requests.
- Additional requests that allow the BSP server to further customize behavior but that aren’t required.

### Standard BSP requests

- `build/initialize` Continue to use the standard request to initialize the connection like we are today, including sending `indexStorePath` and `indexDatabasePath` in the `data` field from the BSP server to SourceKit-LSP, like we do today.
- `buildTarget/inverseSources`: Request from SourceKit-LSP to BSP to get the targets a file belongs to.
  - Caching: The result of this request is assumed to be stable and cashable until the BSP server sends any `buildTarget/didChange` notification to SourceKit-LSP. Every `buildTarget/didChange` notification needs to invalidate the cache since the changed target might gained a source file.

- `buildTarget/sources`: Request from SourceKit-LSP to the BSP server to get the sources in a target, including the following optional extension:
  - BSP servers can set the optional `language?: LanguageId` field in the `SourcesItem` to specify the language that should be used for background functionality. Otherwise, the language is inferred from the file extension.
  - Caching: The result of this request is assumed to be stable and cashable until the BSP server sends a `buildTarget/didChange` to SourceKit-LSP for this target.

- `buildTarget/didChange`: Notification from BSP to SourceKit-LSP to notify that a build target has changed. This causes the target’s file list and the build settings for open files in the target to be reloaded. As an extension, we allow `changes` to be `null` to indicate that all build targets have changed. This is useful for build systems that don’t have target-level tracking of changes, eg. after a `Package.swift` was reloaded in SwiftPM.
- `workspace/buildTargets`: Request to get all targets in the project.
  - BSP servers should set the `"test"` tag for targets that might contain tests so these targets are included in the test navigator.
  - BSP servers can add the custom `"dependency"` tag to a `BuildTarget` to exclude tests from a target from the test navigator. This can eg. be used for test targets of SwiftPM package dependencies.
  - Caching: The result of this request is assumed to be stable and cacheable until the BSP server sends a `buildTarget/didChange` to SourceKit-LSP for any target.

- `window/logMessage`: Notification from the BSP server to SourceKit-LSP that allows logging a message to the editor’s log. SourceKit-LSP forwards this to the editor.

Note that build systems that don’t have a notion of targets (eg. you can consider `compile_commands.json` as such a build system), it is acceptable that they only provide a single dummy target for all files.

### BSP Extensions required for SourceKit-LSP functionality

#### `textDocument/sourceKitOptions`

```swift
/** Request from SourceKit-LSP to the BSP server to retrieve the compiler 
  * arguments that should be used to build the given file in the given target */
export interface BuildSettingsParams {
  textDocument: TextDocumentIdentifier;
  target: BuildTargetIdentifier;
}

export interface BuildSettingsResponse {
  /** The arguments that should be passed to a compiler to build this file */
  compilerArguments: string[];
  
  /** The working directory in which the compiler should be run. 
   * If `null`, the file should be built without a working directory. */
  workingDirectory?: string;
}

```

This is the key request that BSP servers need to implement for SourceKit-LSP. It provides the compiler arguments that are used to provide semantic functionality for the file. This was originally proposed in [SourceKit-LSP for other build systems - #12 by blangmuir](https://forums.swift.org/t/sourcekit-lsp-for-other-build-systems/27109/12) but never implemented.

Closest BSP equivalent: [`buildTarget/cppOptions`](https://build-server-protocol.github.io/docs/extensions/cpp#buildtargetcppoptions-request) in the C++ extension of BSP but its structure is quite different because it eg. explicitly specifies `defines` aka. `-D` flags. `textDocument/sourceKitOptions` is more opaque.

Caching: The result of this request is assumed to be stable and cacheable until the BSP server sends a `buildTarget/didChange` for the target passed in this request.

As described above, the current BSP integration pushes build settings from the build server to the client. SourceKit-LSP has been redesigned to be more pull-based. This pull-based model fits better to recent developments in LSP where eg. diagnostics also became a pull request, initiated by the editor, instead of the LSP server pushing diagnostics. We would thus like to deprecate the existing `build/sourceKitOptionsChanged` notification from the BSP server to the client in favor of this pull model.

### BSP Extensions required for Background Indexing

#### `buildTarget/prepare`

```swift
/** Build Swift modules for this target so that source files in downstream 
 * modules can import modules defined by this target */
export interface PrepareParams {
  /** A sequence of build targets to compile. */
  targets: BuildTargetIdentifier[];
  /** A unique identifier generated by the client to identify this request.
   * The server may include this id in triggered notifications or responses. */
  originId?: OriginId;
}

export interface PrepareResult {}

```

The closest BSP equivalent is `buildTarget/compile` but we have decided to use a different method because preparation doesn’t need to generate any binaries and the generated Swift modules only need to contain declarations, no bodies. It is thus different from a compile.

In the future, we could consider running `buildTarget/compile` if the build server doesn’t support `buildTarget/prepare` in order to try preparing a module. The problem is that a normal compile usually doesn’t continue building if a target’s dependency fails to build but preparation is expected to continue here.

#### `workspace/waitForBuildSystemUpdates`

This is a no-op. If the build system is currently processing updates, like file changes or is currently building the build graph, it should only return once they have finished processing. This is used by background indexing for eg. the current scenario:

- A new source file is added to the project and should thus be indexed in the background.
- We notify the build system about the new file.
- We want to wait until the build system has incorporated the file into its build system so it can provide build settings for it.

### BSP Extensions to help BSP Servers

#### `workspace/didChangeWatchedFiles`

Correctly watching for file changes on all platforms is non-trivial. To help BSP servers that are developed with SourceKit-LSP as its primary user, SourceKit-LSP can forward the LSP notification `workspace/didChangeWatchedFiles` from the editor to the BSP server. To receive the file change notifications, the BSP server must set the `watches: FileSystemWatcher[]` parameter in the initialize response, specifying the glob patterns to watch for.

---

<div class="post-metadata">

**Author:** ![dima\_kozhinov](https://avatars.discourse-cdn.com/v4/letter/d/258eb7/32.png) [@dima\_kozhinov](https://forums.swift.org/u/dima_kozhinov)\
**Post date:** [September 5, 2024, 9:03pm UTC](https://forums.swift.org/t/extending-functionality-of-build-server-protocol-with-sourcekit-lsp/74400/2 "2024-09-05T21:03:16Z")

</div>

Sounds interesting. I'm not familiar with BSP, but I certainly interested in the possibility of SourceKit-LSP supporting alternative build systems besides SwiftPM.

---

<div class="post-metadata">

**Author:** ![rockbruno](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/rockbruno/32/41051_2.png) [@rockbruno](https://forums.swift.org/u/rockbruno)\
**Post date:** [September 9, 2024, 7:58am UTC](https://forums.swift.org/t/extending-functionality-of-build-server-protocol-with-sourcekit-lsp/74400/3 "2024-09-09T07:58:42Z")

</div>

This is great! I have been recently looking into expanding sourcekit-lsp to work with Bazel projects, and it seems this is exactly what I would need in this case.

---

<div class="post-metadata">

**Author:** ![jaredh159](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/jaredh159/32/20920_2.png) [@jaredh159](https://forums.swift.org/u/jaredh159)\
**Post date:** [September 12, 2024, 4:50pm UTC](https://forums.swift.org/t/extending-functionality-of-build-server-protocol-with-sourcekit-lsp/74400/4 "2024-09-12T16:50:28Z")

</div>

very happy to hear about ongoing work on SourceKit-LSP. thank you!

i use neovim as my daily driver, will these changes improve the problem of sourcekit not understanding Xcode projects? i currently architect all my projects to have almost all my code in SPM packages and only thin entrypoints calling out to SPM modules within Xcode projects, because that way i can use neovim with the LSP.

---

<div class="post-metadata">

**Author:** ![skortekaas](https://avatars.discourse-cdn.com/v4/letter/s/9fc29f/32.png) [@skortekaas](https://forums.swift.org/u/skortekaas)\
**Post date:** [September 25, 2024, 2:25pm UTC](https://forums.swift.org/t/extending-functionality-of-build-server-protocol-with-sourcekit-lsp/74400/7 "2024-09-25T14:25:22Z")

</div>

Great to see a proposal for this! I'm mostly curious to get more details on how the `buildTarget/prepare` request will work and how background indexing will be handled in this mode. I understand that you probably don't have the details yet, but I have some high-level questions that come to mind:

- When SK-LSP decides that a file or multiple files need to be reindexed, is the indexing handled entirely by SK-LSP without invoking the build server in any way?
- If SK-LSP does manage index updates independently, I assume the build server returns the path to the index built during `buildTarget/prepare` as part of the response? Or will we set it during `build/initialize`?
- Does `xcodebuild` provide any options that align with the requirements for `buildTarget/prepare`?

Some minor comments:

- In addition to `build/logMessage`, it might make sense to add `build/taskProgress` to allow showing user-facing messages in the editor.
- `workspace/didChangeWatchedFiles` sounds interesting. A colleague from the BSP team mentioned that the Scala language server [Metals](https://github.com/scalameta/metals?tab=readme-ov-file) provides similar functionality by sending progress updates to the build server. Metals seems to have quite advanced build tool integration, so it might be beneficial to take a look at their implementation.

I suggest posting this proposal on [the BSP Discussions page](https://github.com/build-server-protocol/build-server-protocol/discussions) to gather some feedback from the BSP community.

---

<div class="post-metadata">

**Author:** ![ahoppen](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/ahoppen/32/36046_2.png) [@ahoppen](https://forums.swift.org/u/ahoppen)\
**Post date:** [October 2, 2024, 6:45pm UTC](https://forums.swift.org/t/extending-functionality-of-build-server-protocol-with-sourcekit-lsp/74400/8 "2024-10-02T18:45:00Z")

</div>

> [@skortekaas](#):
>
> When SK-LSP decides that a file or multiple files need to be reindexed, is the indexing handled entirely by SK-LSP without invoking the build server in any way?

SourceKit-LSP will call into the build system to prepare a target and then handle the reindexing itself. [sourcekit-lsp/Contributor Documentation/Background Indexing.md at main · swiftlang/sourcekit-lsp · GitHub](https://github.com/swiftlang/sourcekit-lsp/blob/main/Contributor%20Documentation/Background%20Indexing.md) contains some information about how indexing works. It might not answer your question directly but probably contains some information you might find interesting.

> [@skortekaas](#):
>
> If SK-LSP does manage index updates independently, I assume the build server returns the path to the index built during `buildTarget/prepare` as part of the response? Or will we set it during `build/initialize`?

`buildTarget/prepare` should not update the index store. SourceKit-LSP will update the index store when it indexes the source files after preparation is done. Background indexing will populate the index store path that was returned by the `build/initialize` response.

> [@skortekaas](#):
>
> In addition to `build/logMessage`, it might make sense to add `build/taskProgress` to allow showing user-facing messages in the editor.

A recent iteration added support for [`window/workDoneProgress/create`](https://github.com/swiftlang/sourcekit-lsp/blob/main/Contributor%20Documentation/BSP%20Extensions.md#windowworkdoneprogresscreate) and [`$/progress`](https://github.com/swiftlang/sourcekit-lsp/blob/main/Contributor%20Documentation/BSP%20Extensions.md#progress) to show progress in the editor. Does that satisfy your needs?

> [@skortekaas](#):
>
> `workspace/didChangeWatchedFiles` sounds interesting. A colleague from the BSP team mentioned that the Scala language server [Metals](https://github.com/scalameta/metals?tab=readme-ov-file) provides similar functionality by sending progress updates to the build server. Metals seems to have quite advanced build tool integration, so it might be beneficial to take a look at their implementation.

Do you happen to have a link to the Metals’ BSP extensions and how it communicates file changes? I couldn’t find any.

Also, the sourcekit-lsp repo now contains documentation for the [BSP Extensions](https://github.com/swiftlang/sourcekit-lsp/blob/main/Contributor%20Documentation/BSP%20Extensions.md) used by SourceKit-LSP and a [basic guide](https://github.com/swiftlang/sourcekit-lsp/blob/main/Contributor%20Documentation/Implementing%20a%20BSP%20server.md) of the steps needed to implement a BSP server. If you have feedback on the integration, I would like to hear about it.

---

<div class="post-metadata">

**Author:** ![skortekaas](https://avatars.discourse-cdn.com/v4/letter/s/9fc29f/32.png) [@skortekaas](https://forums.swift.org/u/skortekaas)\
**Post date:** [October 25, 2024, 1:51pm UTC](https://forums.swift.org/t/extending-functionality-of-build-server-protocol-with-sourcekit-lsp/74400/9 "2024-10-25T13:51:58Z")

</div>

Thanks for the clarifications. I read the document you added and it looks good to me. One thing I'm not sure about is the use of `window/workDoneProgress/create` and `$/progress`. I think the server should notify the client using `build/taskStart`, `build/taskProgress`, `build/taskFinish`. Where the type of tasks that can be reported is not limited to build-related tasks (see [Build Server Protocol | Build Server Protocol](https://build-server-protocol.github.io/docs/specification#onbuildtaskstart-notification)). The LSP can then propagate these notifications to the editor using `window/workDoneProgress/create` and `$/progress`. See `scala.meta.internal.metals.clients.language.ForwardingMetalsBuildClient` in Scala Metals for an example.

Regarding file watching in Metals, I think I might have misunderstood their approach. I browsed the code and it looks like the client (Metals LSP) doesn't directly send file change notifications to the server. It merely invokes compilation for affected targets when `scala.meta.internal.metals.MetalsLspService#didChangeWatchedFiles` is invoked in the client. I think the approach you took with `workspace/didChangeWatchedFiles` is better.

---

<div class="post-metadata">

**Author:** ![ahoppen](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/ahoppen/32/36046_2.png) [@ahoppen](https://forums.swift.org/u/ahoppen)\
**Post date:** [October 25, 2024, 6:52pm UTC](https://forums.swift.org/t/extending-functionality-of-build-server-protocol-with-sourcekit-lsp/74400/10 "2024-10-25T18:52:24Z")

</div>

Using `build/taskStart` etc. for status reporting instead of the BSP extensions for `window/workDoneProgress/create` and `$/progress` does seem like a good idea. I’m not sure how I missed them. I opened [Use `build/taskStart`, `build/taskProgress` and `build/taskFinish` to communicate progress from a BSP server to the client · Issue #1783 · swiftlang/sourcekit-lsp · GitHub](https://github.com/swiftlang/sourcekit-lsp/issues/1783) to switch that, thanks for the suggestion!

---

<div class="post-metadata">

**Author:** ![buttler](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/buttler/32/37380_2.png) [@buttler](https://forums.swift.org/u/buttler)\
**Post date:** [November 8, 2024, 5:28am UTC](https://forums.swift.org/t/extending-functionality-of-build-server-protocol-with-sourcekit-lsp/74400/12 "2024-11-08T05:28:30Z")

</div>

> [@ahoppen](#):
>
> Using `build/taskStart` etc. for status reporting instead of the BSP extensions for `window/workDoneProgress/create` and `$/progress` does seem like a good idea. I’m not sure how I missed them. I opened [Use `build/taskStart`, `build/taskProgress` and `build/taskFinish` to communicate progress from a BSP server to the client · Issue #1783 · swiftlang/sourcekit-lsp · GitHub](https://github.com/swiftlang/sourcekit-lsp/issues/1783)[applebees](https://applebeeshappyhours.us/) to switch that, thanks for the suggestion!

Thankyou for giving solution. Your answer is worked.