Hi Marc,
I've been thinking about this a bit, along with other build-system-integration questions in relation to SourceKit-LSP. There have been some recent conversations about build system integration in the context of SourceKit-LSP, such as SourceKit-LSP for other build systems - #12 by blangmuir.
My preference would be for a design where the "build system" lives outside of sourcekitd, but sourcekitd can provide the compiler support that's needed to generate swiftmodules (and generated headers). Keeping the build system separate makes it easier for us to share context between sourcekitd and another language server like clangd, and lets sourcekitd focus on its core task of interfacing to the swift compiler.
Here's a sketch of what I have in mind:
- A higher level service such as SourceKit-LSP acts as the build system for sourcekitd. The implementation of this could be partly delegated to a build-system-specific plugin that can talk to your underlying build tools, and maybe cache results to avoid expensive requests.
- The build system service is responsible for (re)generating the inputs necessary for a request in a particular file, including the .swiftmodules
- This could be implemented by shelling out to swiftc if you have local file system access
- Or it could be implemented by calling back to sourcekitd, with a (hypothetical) request that writes a swiftmodule to memory, which we can use with the VFS layer to make available to other sourcekitd requests.
In a model like this, you could imagine also generating a "-Swift.h" header in-memory with sourcekitd and then feeding the header to clangd so that e.g. objc code could see the changes from upstream swift modules.
Another advantage of this model is that the build system has the flexibility to provide other kinds of build intermediates that sourcekitd has no way to generate itself. For example, if your build process generates swift files from gyb, you could theoretically teach the build system service how to do that in-memory and feed that back to sourcekitd as appropriate.
I haven't thought through the details of how you would do this generation in-memory - for example, whether it is a blob of data returned from the request that is then sent back to sourcekitd, or if it just gets written directly to an in-memory vfs in sourcekitd. I also haven't though through whether sourcekitd needs to be able to request the generation of these intermediate inputs, or if this is something a higher-level service should do.
Thoughts?
[0] Nothing here is specific to sourcekit-lsp itself; any higher level service could play the role of build system to sourcekitd.