There is a new post on the Swift.org blog — "Introducing the sourcekitd Stress Tester" — that talks about how a stress tester for SourceKit flushes out issues by systematically running SourceKit queries against the projects in the Source Compatibility Suite.
It can also be used to run on any project to find bugs in sourcekit and the compiler.
If you have questions about the material in the post, please post them here!
When it's said that the stress tester has a long run time, what are we talking about here? Minutes, hours?
Well, testing on my small, incomplete app HazeLight increased the build time by a factor of 14 and produced SR-9877.
I think this'll be a great tool for a critical part of Swift's tooling. Nice job!
I haven't taken precise measurements, but In terms of total project build time I think I've seen up to ~80X before, unfortunately. It stops as soon as it finds an issue though, so on the plus side the longer it takes the less buggy sourcekitd should be in the next release 
Sourcekitd's semantic requests are implemented based on the type-checked AST of the source file, so each time the stress tester modifies a document, the next semantic request causes sourcekitd to re-parse and type-check it to get an up-to-date AST. With the current approach the stress tester takes to modifying the open document, this means the runtime for a particular input source file is roughly proportional to the time it takes to type-check that file * the number of identifiers in it, which is why the overhead is so high.
Thanks a lot for filing!
Does that mean the stress tester could be used to improve sourcekitd performance? Or is the overhead primarily in the generation of the stress cases?
The major overhead is in the sourcekitd functionality not the stress-tester. Something we have in mind is that we could build on top of the stress-tester a mechanism to gather performance metrics (e.g. how long each code-completion invocation took), so we can do general analysis and discover problematic edge cases.
Does this mean we will eventually get to the same speed and reliability when it comes to syntax highlighting and code completion as objective-c code offers?