Background:
I'm working on adding SourceKit to Google's internal source tooling. The architecture is a server that runs one instance of SourceKit per user, and uses those instances of SourceKit to serve requests from IDEs.
To make this work on code that imports modules, I need to give SourceKit access to the appropriate ".swiftmodule" files.
One way to get these ".swiftmodule" files would be to have my server call out to Google's (remote, distributed) build system, which would produce these files. There are some obstacles to doing this:
- The build system is a shared resource that is inappropriate to call every time the user makes a local modification to a module's source.
- There is a pretty big startup latency for each call.
- The Swift compiler on the build system is built and released separately from my SourceKit server, so the binary ".swiftmodule" files will often be incompatible.
Therefore, I'm thinking of making the server itself build necessary ".swiftmodule"s from sources. The server has access to a dependency graph of modules and access to the compiler flags for each module (all this information comes from the build system). The server also has access to all the source files. So I would need to implement some sort of "lite build system" that uses this information to trigger ".swiftmodule" (re)compilations when necessary.
Q1: Does a "lite build system" sound like a reasonable approach?
Are there any drawbacks I should consider? Any better ways to get my server working on code with dependencies?
Q2: Building .swiftmodules in memory
A constraint on my server is that I can't read or write to the local filesystem. So I can't simply spawn "swiftc" processes to build things.
Are there some Swift compiler functions that I can call from my server binary that let me do most of what "swiftc" would do, except that would read inputs from a custom VFS and write inputs to memory?
Inspired by the subcompilation in ParseableInterfaceModuleLoader.cpp, I was able to prototype a working implementation like this:
- Construct a
CompilerInstance ci
and set its filesystem to my custom vfs. - Construct a
CompilerInvocation invok
and set it up usingswift::driver::getSingleFrontendInvocationFromDriverArguments
andinvok.parseArgs
. ci.setup(invok)
ci.performSema()
- Set the module's serialize action to
serializeToBuffers
. (I had to add a new flag to serializeToBuffers that teaches it not to write to the filesystem at all). -
ci.performSILProcessing()
.
Does that seem like a reasonable robust approach?
Q3: Putting this in master & collaborating
Ideally I'd put the "building .swiftmoudles in memory" and the "lite build system" in master with tests so that Swift CI can keep it working and so that others can use and improve it.
Is anyone working on something similar? Would this be useful for anyone else?
Q4: Can we do something faster than compiling .swiftmodules?
It seems that in principle, SourceKit shouldn't need you to fully compile imported modules. It just needs to see the public interface of imported modules, which could be quickly parsed out of a module much faster than the module can be compiled.
Is this something that could be done eventually? Has anyone contemplated doing this?