However, I was using these flags to pre-generate my index-store db considering the size of my index store being decently big (in few gigabytes territory). What I used to do was have a script with some jsonrpc content with a very basic initalize request and pipe it to the sourcekit-lsp with the index db and index store path specified. This allowed me to generate the index-db before I used a new VM for my development. Otherwise, the first 5~10 min of my experience is severely hindered while sk-lsp is writing out the index-db.
Is there a better way to generate the index-db beforehand, now that this flag no longer exists for sk-lsp?
Yes, it’s absolutely still possible to pre-generate the index. We are just inferring the index store from the build system instead of having an option to manually specify it to avoid ambiguities when the compiler arguments could use a different index store path than what is manually specified.
Which build system integration are you using? compile_commands.json, BSP server or a SwiftPM project? For compile_commands.json, ensure that the build settings have an index-store option that matches your index store path, for BSP return the index store path in the build/initialize request as documented here and for SwiftPM projects, I would suggest populating the index store directory inside .build.
Normally for the main lsp functionality on our IDE, we do have a BSP to initialize and set the path accordingly, and this is useful because these paths can differ based on the environment. Because the paths can differ based on the environment, our behavior was to just spin up a sourcekit-lsp through cli that was already built in the preparation stage and piping up some simple lsp init req and keep the connection open for a bit, which allowed sourcekit-lsp to generate the index-db.
However, we can no longer do that, and its not as trivial anymore to override the path when running things from cli alone. Lmk if that doesn't make sense, and if it would make sense to bring the arguments back?
Could you not pass that configuration option into your BSP server and have it return the index store path that you want? I feel like that’s a cleaner solution anyway because it gives the BSP server the option to also use that index store path in the -index-store-path compiler arguments that it sends to SourceKit-LSP. The assumption is that the index store path returned by the BSP server and the index store path in the command line arguments match, especially if you want to support background indexing.
so its a bit complicated, but we are using sourcekit-lsp that was built with swiftpm. To make it work in certain environments / IDE, we use a BSP to launch sourcekit-lsp with the build initialize argument as you outlined. This means that whenever sourcekit-lsp is initialized and used from the IDE experience (vscode, for example), then we can get the correct db / store path. However, if I were to just try to invoke the sourcekit-lsp bin from the cli as sort of like a headless client, I don't know where the set index store / db paths are, and am unable to specify it no longer to pre-generate the index-db before a user kicks it off via the IDE experience, which utilizes the BSP.
Let me maybe rephrase my question: In the workspace that you are opening inside SourceKit-LSP (ie. the folder that is sent in the initialize request), does that contain a Package.swift file, a compile_commands.json, a .bsp folder or a buildServer.json file?
The fact that SourceKit-LSP was built using SwiftPM should not matter here.
Sorry, I may have been unclear with the process (+ took multiple times reading through the code to make sure I understood it fully) and explained it poorly
From my understanding, the current setup is that the vscode extension launches a bsp server as well as sourcekit-lsp. The extension writes the buildServer.json, and upon succesful connection, sourcekit-lsp gets the build settings.
Trying to do this via a simple cli command doesn't seem as straight forward, or am I missing something here?
It’s still unclear to me how your setup works and what exactly you’re trying to achieve. Let’s maybe take a couple steps back:
If you aren’t doing something special, when we are using a BSP server in SourceKit-LSP, SourceKit-LSP looks for a buildServer.json (or alternatively .bsp directory) and launches it based on that configuration. So the pipeline is that a user crafts their buildServer.json, the Swift VS Code extension launches SourceKit-LSP and SourceKit-LSP launches the BSP server that’s been specified in buildServer.json.
Now, I’m not entirely sure which components you have customized:
Are you using a publicly available BSP server or do you have your own?
In your original post, you said that you launch SourceKit-LSP to pre-generate the index. I assume that means that your BSP server supports background indexing and you just send the LSP initialize requests followed by a workspace/synchronize (previously workspace/_pollIndex) request to wait for background indexing to finish? Or am I wrong here?
You say that you build SourceKit-LSP from source. Do you have any customizations to SourceKit-LSP that are relevant?
When you talk about the VS Code extension, do you mean the official Swift extension or something else?
How did you use to specify the -index-store-path command line argument in the past? Did you use the swift.sourcekit-lsp.serverArguments setting in the Swift VS Code extension?
And maybe one potential solution: Could you specify your custom index store location in the buildServer.json’s argv option? That way you can pass the index store path into the build server and it can return that path in the build/initialize response?
We have our own bsp server and have our own background indexing done with our custom logic, but no patches were made to the actual source code to get this accomplished.
In your original post, you said that you launch SourceKit-LSP to pre-generate the index.
Sorry I wasn't clear, the index store is pre-generated already without invoking sourcekit-lsp through other means. We were pre-generating the index-store database through the cli by just kicking up the sourcekit-lsp binary and piping in a basic lsp init request to it. Doing that allowed sourcekit-lsp to find the index store files in the provided index-store path, and then generating the index-store db to the path given as an argument in the cli. Nothing special was required from us.
Do you have any customizations to SourceKit-LSP that are relevant?
Nope
When you talk about the VS Code extension, do you mean the official Swift extension or something else?
Own internal extension
How did you use to specify the -index-store-path command line argument in the past? Did you use the swift.sourcekit-lsp.serverArguments setting in the Swift VS Code extension?
When we started using this, the command line argument existed, so I never had to specify it through different means.
And maybe one potential solution: Could you specify your custom index store location in the buildServer.json ’s argv
I dont think this works because spinning up sourcekit-lsp through our cli method doesn't require / go through a bsp? BSP connection only happens when sourcekit-lsp is kicked through the extension, but not when we just call the sk-lsp binary
I’m a little confused by this. You’re saying that you pre-generate the index without running SourceKit-LSP but then you mention kicking the sourcekit-lsp binary.
The VS Code extension (and thus probably also your internal extension) launches SourceKit-LSP. SourceKit-LSP then looks for for a .bsp folder (or `buildServer.json) in your workspace and launches the BSP server as it is specified in there. Unless you are doing something differently in your extension, the extension does not launch a BSP server, SourceKit-LSP launches it.
Now, assuming that you have .bsp or buildServer.json in your workspace, you can add additional elements to argv. These will be passed to the BSP server when SourceKit-LSP launches it. For example you could add "-index-store-path", "/path/to/your/index/store" to argv here. Your BSP server can then parse that command line argument and return it as the indexStorePath in the build/initialize response and derive an indexDatabasePath relative to that index store path.
These will be passed to the BSP server when SourceKit-LSP launches it. For example you could add `"-index-store-path"
If I'm understanding this correctly, assuming sourcekit-lsp bin is located in dir A, if there is a builserver.json or .bsp in that dir and I just try to spin up a sourcekit-lsp bin through the cli, it will autmatically look for that bsp?
I'm still a bit lost, because my understanding is that just calling sourcekit-lsp bin through the cli doesn't seem to do any interaction with the bsp. Is that not the case?
If you have a buildServer.json or a .bsp folder in your workspace, SourceKit-LSP launches it to get the build settings it uses for semantic functionality.
Hey Alex! Sorry for getting back on this late. Finally got some time to try this out, and was unable to get it work successfully
So in my case with the regular workflow (kicking sourcekit-lsp off through the editor extension:
SourceKit-LSP reads buildServer.json and launches a build server proxy using the argv command specified
SourceKit-LSP sends a build/initialize request to the build server proxy over the Unix domain socket
The build server proxy responds with an InitializeBuildResult that includes the index paths in the data field
However, I'm not entirely sure how to ensure buildServer.json is in my workspace / gets picked up when launching sourcekit-lsp just through the cli. Going through the above process just to build out the index-db is far more complex than just shelling out a simple init request like I used to directly to the sourcekit-lsp, so I'd like to avoid that if possible. It sounds like it should be possible if it picks up the buildServer.json when I'm just running the sourcekit-lsp executable, but I've tried creating a buildServer.json in my normal workspace root, where sourcekit-lsp executable is, etc etc all to no avail. Does the json need to be piped in to sklsp somehow?
However, I'm not entirely sure how to ensure buildServer.json is in my workspace / gets picked up when launching sourcekit-lsp just through the cli.
All the steps you mentioned happen inside sourcekit-lsp itself. Tthe editor is just sending an initialize request to the same CLI, which sounds like the same thing you were doing in:
shelling out a simple init request like I used to directly to the sourcekit-lsp