Developing a sourcekit-lsp Client: Why Won't it Talk to Me? 🥺

[flowtoolz (Sebastian Fichtner)]

Ben has already been very helpful recently with some preparing tasks

Now, I haven't yet managed to get useful responses back from sourcekit-lsp, and after days of trying to narrow down the issue, I want to kindly ask for help again ...

The client sends an initialize request and gets server capabilities back. (The initialize request has in params.rootUri an absolute file path to a Swift package folder.)

Then the client gets 2 notifications of type 4 and with these messages:

• manifest parse error(s):\nThe operation couldn’t be completed. (TSCBasic.Process.Error error 0.)
• failed to create SwiftPMWorkspace: the package does not contain a buildable target

However, the Swift packages I tried are valid and can be built from Xcode and console and do have buildable targets.

After that, no request works: The textDocument/didOpen request produces the error LSP response method not found. And textDocument/documentSymbol and workspace/symbol requests return successful responses with empty results.

My main question: Do the above LSP notifications indicate an underlying error that would prevent the mentioned requests to work?

Because I know so little, I can imagine (and have tried) a million levers that might be relevant here ... Btw, the picture is the same when I use SwiftLSPClient, so my client side implementation can't be the only problem.

A few "sub-questions":

• Does the sourcekit-lsp process need to be launched with specific option arguments to make the above requests work?
• Does the process need a specific environment?
• Does the initialize request need to have specific capabilities or other properties?
• And generally: Are there any resources out there aside from the LSP specs and sourcekit-lsp itself that help with implementing a sourcekit-lsp (or other LSP-) client?

Gonna take a break and practice patience now, as I can't force the project forward without help. I'm grateful for any hints that alleviate my ignorance

[blangmuir (Ben Langmuir)]

@flowtoolz

manifest parse error(s):\nThe operation couldn’t be completed. (TSCBasic.Process.Error error 0.)

I'm not sure what's going on here. Do you see the same behaviour with all packages (e.g. fresh swift package init)? Are you using sourcekit-lsp from the main branch, and if so what swift toolchain are you using with it?

After that, no request works: The textDocument/didOpen request produces the error LSP response method not found .

Just a guess, but are you sending it as a "request"? This message is a notification, so it should not have a request id (specification). I filed SR-13763 to improve the error message for this situation.

If that's not the problem, could you send the exact message so I can look into it?

And textDocument/documentSymbol and workspace/symbol requests return successful responses with empty results.

This is happening because the document isn't open, because of the previous failure. We should improve this to return an error or make it use the file from disk. I filed a bug SR-13762.

• Does the sourcekit-lsp process need to be launched with specific option arguments to make the above requests work?

No. You could have a situation where a swiftpm package cannot be loaded correctly without additional options, but this should only happen if you also need to pass additional options on the swift build command line, and it would not have the symptoms you described above (except maybe the one about the manifest parse issue).

• Does the process need a specific environment?

No.

• Does the initialize request need to have specific capabilities or other properties?

No. Specific requests may be impacted if there are related client capabilities missing, but none of the things you described above are like that.

• And generally: Are there any resources out there aside from the LSP specs and sourcekit-lsp itself that help with implementing a sourcekit-lsp (or other LSP-) client?

If you are having trouble with a request and you can trigger it from VSCode (or some other LSP-compatible editor) where it is working, it would be helpful to dump the request json (e.g. sourcekit-lsp.trace.server "verbose" in vscode) and compare it with what your client is doing. I see you mentioned trying SwiftLSPClient, so this is the same idea, but I have no experience with SwiftLSPClient specifically, whereas VSCode is the de facto reference implementation of an LSP client.

[flowtoolz (Sebastian Fichtner)]

@blangmuir

Do you see the same behaviour with all packages (e.g. fresh swift package init )?

Yes. All the same.

@blangmuir

Are you using sourcekit-lsp from the main branch, and if so what swift toolchain are you using with it?

Everything I wrote here holds true for Xcode 12.1 and launching via xcrun sourcekit-lsp. But I've tackled this also with Xcode 12.0.1 and latest main branch release builds of sourcekit-lsp.

@blangmuir

but are you sending it as a "request"? This message is a notification

Oh you're right, I mistakenly had this in the request extension. Sending textDocument/didOpen as a notification now provokes this notification from the server: could not open compilation database for \/Users\/seb\/Desktop\/TestProject\/Sources\/TestProject\/TestProject.swift

@blangmuir

If that's not the problem, could you send the exact message so I can look into it?

Initialize request (already produces the manifest/workspace errors):

Content-Length: 214

{
  "jsonrpc" : "2.0",
  "id" : "DC649CF3-F767-4DE7-9A83-C0E467652089",
  "method" : "initialize",
  "params" : {
    "capabilities" : {

    },
    "rootUri" : "file:\/\/\/Users\/seb\/Desktop\/TestProject\/"
  }
}

Did open notification:

Content-Length: 340

{
  "jsonrpc" : "2.0",
  "method" : "textDocument\/didOpen",
  "params" : {
    "textDocument" : {
      "text" : "struct TestProject {\n    var text = \"Hello, World!\"\n}\n",
      "languageId" : "swift",
      "version" : 1,
      "uri" : "file:\/\/\/Users\/seb\/Desktop\/TestProject\/Sources\/TestProject\/TestProject.swift"
    }
  }
}

@blangmuir

This is happening because the document isn't open, because of the previous failure.

Just to understand: The specs say the document doesn't need to be open for the server to respond to requests. I assume without a [failing] didOpen notification, the server would read the document from the file system to provide the document's symbols.

And is it the case that sourcekit-lsp must create the SwiftPMWorkspace to work properly? Then that might be the problem root and everything else more or less noise?

@blangmuir

If you are having trouble with a request and you can trigger it from VSCode (or some other LSP-compatible editor) where it is working, it would be helpful to dump the request json (e.g. sourcekit-lsp.trace.server "verbose" in vscode) and compare it with what your client is doing.

Ah nice! That's another option Thanks a lot!

[blangmuir (Ben Langmuir)]

@flowtoolz

Everything I wrote here holds true for Xcode 12.1 and launching via xcrun sourcekit-lsp . But I've tackled this also with Xcode 12.0.1 and latest main branch release builds of sourcekit-lsp.

Just to confirm: did you build the package using the same toolchain? The build should match the toolchain you're using sourcekit-lsp from. I'm not seeing any problems loading packages in sourcekit-lsp from Xcode 12/12.1.

could not open compilation database for

This is a consequence of the swiftpm failure. Since it couldn't load a swiftpm workspace it is trying to fall back to looking for a compilation database.

Just to understand: The specs say the document doesn't need to be open for the server to respond to requests. I assume without a [failing] didOpen notification, the server would read the document from the file system to provide the document's symbols.

The spec says we may do that, not that we must. Some servers are even designed to run in environments where they have no filesystem access. But yes, I think that ideally we would load it from disk when it's not open. It will just take a bit of work, since we need to handle transitioning from the on-disk state to the in-memory state and back again correctly.

And is it the case that sourcekit-lsp must create the SwiftPMWorkspace to work properly? Then that might be the problem root and everything else more or less noise?

If the swiftpm workspace fails to load, it means we will not get compiler arguments for your files. Features will still "work" in the sense that we will make a best effort, but they will have a single-file view on your code. If you were editing a script file, that would probably be fine. But in a real package you will get errors about things defined in other files or other modules, and code-completion and hover will not know about those symbols, definition will not find anything outside the current file, etc.

[flowtoolz (Sebastian Fichtner)]

@blangmuir

Just to confirm: did you build the package using the same toolchain? The build should match the toolchain you're using sourcekit-lsp from.

I reset the toolchain path, deleted old Xcodes, rebuilt the test project, rebuilt sourcekit-lsp, ensured the client uses the freshly built sourcekit-lsp and ... that solved it

This poses tons of questions for me ... Maybe you have a rough stroke answer.

Btw, I'd like to extend the sourcekit-lsp documentation to adress client developers, if that's ok, so I'm not the only one benefiting from my 2 threads on this, but for that I'd need to understand this better.

So, correct me if I'm wrong: One can't use the sourcekit-lsp binary that comes with Xcode 12.1 on projects built with Xcode 12.1? What can that "official" version be applied to?

Took me a minute to get what you meant with "the package". I thought why is it important what Xcode version I used last time I built the package? Why is it important at all that the package has already been built? (If that's a fair way to articulate this) Does sourcekit-lsp rely on some intermediate "byproducts" of the build process?

So then this could never work on package folders that contain no build artifacts, for example on remote GitHub repos, or on packages exclusively built from Xcode where build artifacts land in the derived data folder?

How would a client recognize which Swift package requires which sourcekit-lsp version and where should it get the different sourcekit-lsp versions from?

Or is all this just due to a still "unofficial" stage of sourcekit-lsp integration in Xcode?

@blangmuir

Just to understand: The specs say the document doesn't need to be open for the server to respond to requests. I assume without a [failing] didOpen notification, the server would read the document from the file system to provide the document's symbols.

The spec says we may do that, not that we must. Some servers are even designed to run in environments where they have no filesystem access.

That makes total sense and would allow sourcekit-lsp to be offered as a web service. Although it seems in conflict with the above discussed need to access build artifacts.

The specs of the did open notification do confuse me though. They state:

This means open and close notification must be balanced and the max open count for a particular textDocument is one. Note that a server’s ability to fulfill requests is independent of whether a text document is open or closed.

I see 2 change requests coming out of this for sourcekit-lsp:

• As you suggested: Sending error response instead of empty result on symbol requests for closed documents.
• Sending an error response (or any explicit form of error feedback) instead of log message notifications, when the SPM workspace can't be instantiated on initialize request.

Thank you so much for helping me close my sourcekit-lsp development "feedback loop". I can now plow ahead

[blangmuir (Ben Langmuir)]

@flowtoolz

So, correct me if I'm wrong: One can't use the sourcekit-lsp binary that comes with Xcode 12.1 on projects built with Xcode 12.1? What can that "official" version be applied to?

This works if they're both using 12.1. Did you mean for one of these to be a different version? What is currently supported is using a single toolchain for both building and sourcekit-lsp.

@flowtoolz

Took me a minute what you meant with "the package". I thought why is it important what Xcode version I used last time I built the package? Why is it important at all that the package has already been built? (If that's a fair way to articulate this) Does sourcekit-lsp rely on some intermediate "byproducts" of the build process?

Exactly. There are two parts to think about here:

1. The toolchain swift compiler used to build swiftmodules and sourcekitd.framework that loads them in sourcekit-lsp
2. The toolchain swiftpm that creates build intermediate files and the copy of libSwiftPM linked in sourcekit-lsp that loads them

For (1), there is no hard dependency. SourceKit-LSP can use sourcekitd.framework from different toolchains at runtime, so if you want to use sourcekit-lsp from toolchain A and build your code with swiftc from toolchain B, you theoretically just need to change which toolchain sourcekit-lsp uses at runtime, e.g. using the Toolchain Path in VSCode settings.

For (2), SourceKit-LSP links to libSwiftPM so changing the toolchain (e.g. with the Toolchain Path setting in VSCode) cannot change which swiftpm we are using. That means it only works if the build intermediate file formats and their layout on disk is compatible between the toolchain you did the build with and the version linked to sourcekit-lsp. The swiftpm folks don't go out of their way to break this compatibility, so in practice this often works across versions, but it's only guaranteed to work if you use sourcekit-lsp from the same toolchain as what you used to build.

@flowtoolz

So then this could never work on package folders that contain no build artifacts, for example on remote GitHub repos, or on packages exclusively built from Xcode where build artifacts land in the derived data folder?

Not "never", but it's not a straightforward thing to fix. There are two orthogonal ways we have talked about to improve this in the future:

1. Reduce our dependence on the user's build directory in sourcekit-lsp.

   1. We could create a separate build sub-directory for sourcekit-lsp's use and generate the .swiftmodules etc. ourselves instead of sharing them with the user's build so we know they are using a compatible toolchain. The reason we don't already do this is performance. We don't want to kick off a full build of the code. A lot of the compiler work needed to make module generation fast has already been done, but we would still need to implement the build system support for triggering a build of just the minimal dependencies of the file you are opening (probably needs work in both swiftpm and sourcekit-lsp).
   2. Separately from the .swiftmodules we would need access to the indexing data. Since indexing is fundamentally more expensive than generating swift modules, we probably will always want to try to share the data with the build. But we could add support for background indexing so that you are not required to build to get indexing.
   3. There is still a remaining issue, which is the source checkouts of dependencies that also live in the .build directory. Like the indexing data, we would prefer to continue sharing this with the build when possible. For the case where dependencies haven't been checked out yet, sourcekit-lsp should probably not be the one actually downloading them. Either we should inform the IDE to do it for us, or if the IDE does not support that we could send an alert message to the user with instructions to trigger checking out the package dependencies.

2. Provide a stable interface to swiftpm so that we can use the version from the toolchain instead of linking to a particular version.

   • SourceKit-LSP already has support for using an external build server using a stable API. If we moved swiftpm support to use that approach, we could launch a swiftpm build server from whatever toolchain is configured.

Sending an error response (or any explicit form of error feedback) instead of log message notifications, when the SPM workspace can't be instantiated on initialize request.

I'm not sure this is desirable in general. The only obvious place to send this error as anything other than a log message is during initialization, but (a) we don't want to block initialization on loading the workspace for performance reasons, and (b) generally we prefer to provide best effort support rather than failing entirely.

You could add a new server option if you want to change this behaviour only for clients that opt-in, that seems reasonable though.

[flowtoolz (Sebastian Fichtner)]

@blangmuir

This works if they're both using 12.1. Did you mean for one of these to be a different version? What is currently supported is using a single toolchain for both building and sourcekit-lsp.

I built the test package with Xcodes 12.1 and tried to inspect that package with the sourcekit-lsp executable that locates in the default Toolchain of Xcode 12.1. (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/sourcekit-lsp). That did not work. It seems the sourcekit-lsp version within Xcode 12.1 is older (October 8) than that Xcode itself, so "their toolchains" don't match ...

I just need to get used to the fact, that the sourcekit-lsp delivered with Xcode is incompatible with the very Toolchain of which it is a part. So it's "from" that Toolchain, but more importantly, it hasn't been built with it, so it's incompatible

@blangmuir

if you want to use sourcekit-lsp from toolchain A and build your code with swiftc from toolchain B, you theoretically just need to change which toolchain sourcekit-lsp uses at runtime

Definitely good to know that sourcekit-lsp can be pointed to different toolchains!

To recapture: In a standard setup, my user has one Xcode and the one sourcekit-lsp in Xcode's default toolchain. Since both might be incompatible with each other (if above observations are true), the user must either download a toolchain that's compatible with his sourcekit-lsp or build/download a sourcekit-lsp that's compatible with his toolchain.

I'm gonna need to automate all that and hide it from the user or hope that the situation improves one day when Xcode itself relies on sourcekit-lsp Just imagine a code visualizer that shows metrics to managers or an educational app that teaches coding to kids ... Everything just has to work there out of the box

@blangmuir

(a) we don't want to block initialization on loading the workspace for performance reasons, and (b) generally we prefer to provide best effort support rather than failing entirely

(a) is what I also thought and (b) makes total sense, since a single script file, as you mentioned, can still be inspected. That leaves maybe stdErr. Error responses for requests that need the workspace are probably already in place.

Edit: On (a): If the initialize request explicitly has a rootUri, then I it might be worth letting it wait and returning an error, since it has made the intention clear to use the workspace. Also, initialization happens once and is critical. Why would performance concerns need to compromise function.

Thanks again for your detailed insights! When I'll have absorbed the subject and developed against sourcekit-lsp more, I'm gonna make a PR to extend the README, if you don't mind.

[blangmuir (Ben Langmuir)]

@flowtoolz

I built the test package with Xcodes 12.1 and tried to inspect that package with the sourcekit-lsp executable that locates in the default Toolchain of Xcode 12.1. (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/sourcekit-lsp). That did not work.

Within a single toolchain, sourcekit-lsp should always be in sync with swiftpm. Just to be sure, I verified that the sourcekit-lsp/swiftpm versions are properly in sync in Xcode 12.1.

If you're seeing an issue within this single toolchain I don't think it's related to being out of sync with swiftpm. It would help to have a specific set of instructions to reproduce the issue so we can investigate what's happening. I do not see any issues using sourcekit-lsp from 12.1 with the packages I tried.

It seems the sourcekit-lsp version within Xcode 12.1 is older (October 8) than that Xcode itself, so "their toolchains" don't match ...

I'm not sure what you're looking at to see Oct 8, but if that's a timestamp from the file system it doesn't tell you anything about whether they are in sync.

[flowtoolz (Sebastian Fichtner)]

@blangmuir

Within a single toolchain, sourcekit-lsp should always be in sync with swiftpm.

Ah, so this was another red herring and should indeed have worked?

@blangmuir

It seems the sourcekit-lsp version within Xcode 12.1 is older (October 8) than that Xcode itself, so "their toolchains" don't match ...

I'm not sure what you're looking at to see Oct 8, but if that's a timestamp from the file system it doesn't tell you anything about whether they are in sync.

The sourcekit-lsp within Xcode 12.1 (which I run via xcrun sourcekit-lsp) is marked as created on Oct 8 so it's at least that old and cannot have been built with Xcode 12.1 and cannot be the latest sourcekit-lsp that was available when Xcode 12.1 was released. That was my desperate attempt to piece together why it doesn't work on projects built with Xcode 12.1 Only the sourcekit-lsp that I built myself using Xcode 12.1 works ...

First one can't create workspace, second one can:

static var all: [LanguageKey: Config] = [
//  "swift": .init(executablePath: "/usr/bin/xcrun",
//                 arguments: ["sourcekit-lsp"]),
  "swift": .init(executablePath: "/Users/seb/Desktop/sourcekit-lsp-release",
                 arguments: []),
  "python": .init(executablePath: "/Library/Frameworks/Python.framework/Versions/3.9/bin/pyls",
                  arguments: [])
]

I'll play around some more, maybe I can narrow it down better or find a good way to reproduce it ...

[blangmuir (Ben Langmuir)]

@flowtoolz

Ah, so this was another red herring and should indeed have worked?

Correct. The mismatch should only every happen between separate toolchains, or between a toolchain and sourcekit-lsp built from source.

@flowtoolz

The sourcekit-lsp within Xcode 12.1 (which I run via xcrun sourcekit-lsp ) is marked as created on Oct 8 so it's at least that old and cannot have been built with Xcode 12.1 and cannot be the latest sourcekit-lsp that was available when Xcode 12.1 was released.

Just to re-emphasize, none of those things can be inferred from the timestamp on the binary. Without intimate knowledge of how the software was built, packaged, and installed, you cannot interpret the timestamp.

I'll play around some more, maybe I can narrow it down better or find a good way to reproduce it ...

That would be great!