Background indexing for SourceKit-LSP

Hi,

Are there any updates/plans on support for automatic background indexing in sourcekit-lsp?

Thanks,
Anthony

We plan to add it, but it's not an immediate priority. For now, I encourage using indexing while building to produce index data.

last time I checked you were in favor of indexing on the client side, did you change plans?

I'm not sure what you're referring to; are you talking about when we discussed the patch to do background building? I see that as separable from background indexing. Our plan has always been to support indexing-while-building from the start, and eventually add support for background indexing.

Thanks for the prompt response! Having not looked into the SourceKit-LSP codebase before, do you think it would be reasonable to try and contribute this functionality? How big are the set of changes you expect it would require?

I saw found this post which lists some high-level steps. Could someone provide more detailed guidance if I decided to take this on?

I'm refering to this Try to build SPM workspace at open, if index store is not available. by krzyzanowskim · Pull Request #69 · apple/sourcekit-lsp · GitHub
I believe both, building and indexing was discussed in thread

From that link:

The way forward IMO is to have:

  • A way to efficiently only generate swift modules, ideally shared with the build.
  • Have a background indexer (not just do a full build) but in a way where it can be cancelled as soon as the user initiates a build.

In absence of the above I prefer the existing situation over adding a full background build. The most common case is that the user will initiate a build on their own anyway and now you have two builds happening concurrently.

This is basically how I see things as well. We want to have a background indexer that is is not just doing a build.

I might read it wrong there.
do you plan to get it from Xcode, and therefore we should wait for it, or it's something the community may tackle?

I'd like to take this opportunity that as far as indexing is concerned there is an additional third functionality, separate from index-while-building and background-index-changed-files, which is "get the index data for the in-memory source changes".

Essentially what we'd like to have is index data to be generated/updated as you edit the file, with the in-memory source changes. The functionality to generate the index data would be reside in sourcekitd (the compiler side) and controlled by SourceKit-LSP.

Note that this has less scope and complexity than designing and implementing a background indexer, and it makes the background indexer less of a necessity for getting index data from edits without needing to built.

That does sound like it would cover “live” indexing which would be great. @akyrtzi could you please give a brief summary of what changes that would require assuming I know pretty much nothing about SourceKit? :)

This should be a new sourcekitd request that works similarly to cursor-info in that it reuses an existing, AST if it is still valid, and it uses the AST to generate the index data similarly to how the compiler is doing it when -index-store-path is used.

Note that there is an existing index request currently but this is an old and unused request since we introduced the new index-while-building system. It also always creates a new AST instead of re-using the existing one.

If you'd like to debug sourcekitd and step through how cursor-info works you can see how to invoke it via sourcekitd-test from the tests.