Hey all,
I have been working on filling in gaps in our testing story for SourceKit-LSP and IndexStoreDB, and I wanted to advertise the new test support code for anyone who is developing for SourceKit-LSP. Going forward, all changes should come with appropriate tests.
The major hole in our unit tests for SourceKit-LSP and IndexStoreDB up until now has been that we had no way to get index data for a test project so that we could test (a) the index itself, and (b) the use of the index within SourceKit-LSP, for example the textDocument/references request.
My goals in working on test support were to make it possible to easily and efficiently,
- Define a test project (fixture)
- Build .swiftmodules and index data
- Modify sources and incrementally rebuild
- Provide compiler arguments to SourceKit-LSP
Test Projects and Tibs Build System
Tests can now use shared test projects (fixtures) written in the Tests/INPUTS directory. They use a simple JSON manifest to describe their targets and source files.
Tests/
INPUTS/
MyTestProj/
project.json // contents: { "sources": ["a.swift", "b.swift"] }
a.swift
b.swift
In order to support efficiently building .swfitmodules and index data for such projects, I wrote a simple build system called "Tibs" (the "Test Index Build System") that parses the manifest and writes a Ninja build description. This enables writing multi-module, mixed language projects, and minimizes the work done to produce only the outputs we need, since we can skip emitting object code, etc. Building on top of Ninja allows us to get correct incremental builds, which we also leverage to avoid rebuilding test projects unnecessarily during local development. Only tests that mutate their sources need to be rebuilt in most cases.
Why not use the Swift Package Manager?
The primary reason not to use the Swift Package Manager (SwiftPM) for all of our test projects is that SwiftPM's model is stricter than other build systems we need to support, and stricter than we want for our testing support. For example, we want to be able to test mixed language targets (using bridging headers), and to perform only the module-generation and indexing parts of the build without emitting object code. We need to be able to add features to our test support that would break the clean model that SwiftPM provides.Source Locations
It is common to want to refer to specific locations in the source code of a test project. This is supported using inline comment syntax.
Test.swift:
func /*myFuncDef*/myFunc() {
/*myFuncCall*/myFunc()
}
In a test, these locations can be referenced by name. The named location is immediately after the comment.
let loc = ws.testLoc("myFuncDef")
// TestLocation(url: ..., line: 1, column: 19)
There are APIs for converting TestLocations to the currency types in the index and SourceKit
let loc = ws.testLoc("myFuncDef")
let occurrence = Symbol(...).at(loc, roles: .definition)
let lspLocation = Location(loc)
let lspPosition = Position(loc)
Test Workspaces
To manage all the pieces needed for loading, building and working with test projects, there is a new SKTibsTestWorkspace (or TibsTestWorkspace in IndexStoreDB). This takes care of setting up the build and any temporary directories for e.g. the index, as well as providing convenience APIs for building.
func testFoo() {
// Create the workspace, including opening a connection to the TestServer.
guard let ws = try staticSourceKitTibsWorkspace(name: "MyTestProj") else { return }
let loc = ws.testLoc("myLocation")
// Build the project and populate the index.
try ws.buildAndIndex()
// Open a document from the test project sources.
try ws.openDocument(loc.url, language: .swift)
// Send requests to the server.
let response = try ws.sk.sendSync(...)
}
Documentation
There are more details available in the guides to writing tests in the individual repos: