# \[SR-710\]\[RFC\] Automatically detecting XCTest test methods on Linux: Reflection? SourceKit?

**URL:** <https://forums.swift.org/t/sr-710-rfc-automatically-detecting-xctest-test-methods-on-linux-reflection-sourcekit/2050>\
**Category:** Compiler\
**Created:** [April 3, 2016, 9:11pm UTC](https://forums.swift.org/t/sr-710-rfc-automatically-detecting-xctest-test-methods-on-linux-reflection-sourcekit/2050 "2016-04-03T21:11:28Z")\
**Posts on this page:** 1\
**Showing post:** 12

<div class="post-metadata">

**Author:** ![ddunbar](https://sea2.discourse-cdn.com/swift/user_avatar/forums.swift.org/ddunbar/32/1933_2.png) [@ddunbar](https://forums.swift.org/u/ddunbar)\
**Post date:** [May 6, 2016, 8:04pm UTC](https://forums.swift.org/t/sr-710-rfc-automatically-detecting-xctest-test-methods-on-linux-reflection-sourcekit/2050/12 "2016-05-06T20:04:19Z")

</div>

Sorry for the delay in following up.

I have had several long discussions on this topic with Dmitri and others, which  
I will try to summarize here:

\*TL;DR\*: We think the right long-term path forward is to pursue porting  
SourceKit to Linux, and would like to explore that direction first before trying  
to develop a compromise which either (a) uses internal and likely-to-break APIs  
(e.g., dump-ast) or (b) devotes significant engineering resources to a "right  
solution" for this problem, but which will only solve this problem and not other  
issues where having an API for use with the Swift language.

## Problem Statement

The XCTest API has historically been defined in terms of methods with a  
particular naming convention `test...` which were in subclasses of XCTestCase.

On OS X, these methods can be found via the Objective-C runtime but that does  
not work on Linux. Our current solution on Linux requires manual specification  
of all of the test methods, and is a huge maintenance burden for people trying  
to use XCTest on Linux or maintain cross-platform projects.

## Background

This feature works on OS X in two ways:

1. As mentioned, tests are run by dynamic discovery through the Objective-C  
&nbsp;&nbsp;&nbsp;runtime when the bundle containing the tests is executed.  
&nbsp;&nbsp;&nbsp;  
2. Within the Xcode IDE, tests are "discovered" (for use in UI) through the use  
&nbsp;&nbsp;&nbsp;of the Xcode indexer.  
&nbsp;&nbsp;&nbsp;  
&nbsp;&nbsp;&nbsp;The mechanism at work here, for Swift, is a combination of functionality in  
&nbsp;&nbsp;&nbsp;the indexing engine (to aggregate and query results) and the raw underlying  
&nbsp;&nbsp;&nbsp;data provided by SourceKit.  
&nbsp;&nbsp;&nbsp;  
&nbsp;&nbsp;&nbsp;We would like any implementation of this feature to share as much code as  
&nbsp;&nbsp;&nbsp;possible with SourceKit and Xcode's implementation in order for  
&nbsp;&nbsp;&nbsp;cross-platform projects to behave predictably.  
&nbsp;&nbsp;&nbsp;  
It also happens that SwiftPM has several other needs for API-based interactions  
with the functionality in the Swift compiler. Several examples:  
&nbsp;&nbsp;&nbsp;  
\* We would like to be able to enforce the strictness of the `Package.swift`  
&nbsp;&nbsp;manifest file format. This requires APIs to interact with the Swift  
&nbsp;&nbsp;AST. [[SR-1433] [SwiftPM] Enforce manifest strictness · Issue #5293 · apple/swift-package-manager · GitHub](https://bugs.swift.org/browse/SR-1433)  
&nbsp;&nbsp;  
\* We would like to support advanced features for dealing with automatic  
&nbsp;&nbsp;inter-module dependencies. For example, one idea which has been proposed is  
&nbsp;&nbsp;that if a module attempted to import a module upon which it did not have a  
&nbsp;&nbsp;dependency, that we would prompt the developer if this dependency should  
&nbsp;&nbsp;automatically be added to the manifest. Doing this feature well requires  
&nbsp;&nbsp;APIs to interact with the Swift compiler as it parses source code and  
&nbsp;&nbsp;reports diagnostics.  
&nbsp;&nbsp;  
\* We would like to understand the current level of parallelism being used by  
&nbsp;&nbsp;the Swift compiler, so that llbuild can accurately schedule work. This  
&nbsp;&nbsp;requires APIs for interacting with an in-flight compilation process.

## Discussion

We discussed several avenues of attack on this problem. I will go through them  
one by one. I am just attempting to summarize the conversation, obviously there  
is a ton of nuance to each point, and hopefully other people will chime in if I  
missed something important.

### Language Features

One way to view this problem is that XCTest's API (i.e., depending on test  
naming and dynamic discovery) is a poor fit for Swift today, and that this  
problem should be tackled in that direction.

For example, the Swift compiler itself has a test framework that does not depend  
on dynamic discovery. One could also imagine language-level features which would  
solve the arbitrary problem of wanting to find discoverable things; that could  
take the form of an `@XCTest` attribute, or "generalized attribute" support.

The current mission for Swift 3, however, is API parity between Linux and OS X,  
and so this direction does not lead to any short term solution. For that reason,  
while many of us ultimately think this is the right long term direction, we need  
to find another one as well.

### Custom "Supported" XCTest Feature

The next option is to pursue a custom, but "supported" feature intended to  
tackle this problem head on. Some proposals that have come up are, e.g., a new  
compiler flag which emits the list of test methods.

This approach has a couple unfortunate properties:

1. It is non-trivial. We can either design this as an incredibly XCTest specific  
&nbsp;&nbsp;&nbsp;feature requiring understanding of the backend (compiler directly emits  
&nbsp;&nbsp;&nbsp;metadata into .o file), or a midway feature (compiler tells us list of test  
&nbsp;&nbsp;&nbsp;methods, but then we have to manage incremental compilation and the desire to  
&nbsp;&nbsp;&nbsp;not compile things multiple times more than necessary ourselves).  
&nbsp;&nbsp;&nbsp;  
2. It is not-reusable. The work we do here doesn't help with any of the other  
&nbsp;&nbsp;&nbsp;ways we want to use the Swift compiler as an API.  
&nbsp;&nbsp;&nbsp;  
### Implement an API-based Interface

This approach means porting SourceKit to Linux, and then building this feature  
on top of that (that itself will be non-trivial, which is something not to be  
glossed over).

Everyone generally agrees we should have some kind of API-based access to the  
compiler, so this is in line with our long-term direction, but in a way which is  
actionable now (it does not require design).

This approach has its own issues:

1. Porting SourceKit may require a fair amount of work. The current code base is  
&nbsp;&nbsp;&nbsp;very OS X specific in particular areas (generally around the use of  
&nbsp;&nbsp;&nbsp;`libxpc`). This will require adding abstractions, developing alternatives,  
&nbsp;&nbsp;&nbsp;and writing the code and build system changes to get this to happen.  
&nbsp;&nbsp;&nbsp;  
2. Somehwat unrelated, but the compiler itself (`swiftc`) is not yet written in  
&nbsp;&nbsp;&nbsp;a way that it can be used from SourceKit. If we do port and use SourceKit, it  
&nbsp;&nbsp;&nbsp;will mean the toolchains increase in size because we effectively will have  
&nbsp;&nbsp;&nbsp;two copies of many parts of Swift/Clang/LLVM installed.  
&nbsp;&nbsp;&nbsp;  
3. By itself, this does not fully solve the issue. As mentioned in the  
&nbsp;&nbsp;&nbsp;background, SourceKit itself does not directly manage all parts of XCTest  
&nbsp;&nbsp;&nbsp;discovery today on OS X, it is a collaboration between SourceKit and the  
&nbsp;&nbsp;&nbsp;Xcode indexer to do discovery without execution. The expectation, however, is  
&nbsp;&nbsp;&nbsp;that we can, however, use SourceKit's APIs to implement this feature in a way  
&nbsp;&nbsp;&nbsp;that is "supported".

## Conclusion

The conclusion was that after weighing all of the tradeoffs, it made the most  
sense to encourage porting of SourceKit to Linux and then using it to build out  
the Linux test discovery feature. This was most in line with a desirable  
long-term direction without being blocked on language design.

I don't think we are particularly firm in this conclusion. If you feel that the work  
you have already invested gets us so close that it is worth prioritizing that  
approach, then please push back.

- Daniel

> **···**
>
> > On May 1, 2016, at 8:19 PM, Brian Gesiak \<modocache@gmail.com\> wrote:
> > 
> > I made an attempt at adding `swiftc -frontend -dump-xctest-methods`: [[SR-710][Frontend] Add `-dump-xctest-methods` by modocache · Pull Request #2364 · apple/swift · GitHub](https://github.com/apple/swift/pull/2364)
> > 
> > Feedback appreciated!!
> > 
> > - Brian Gesiak
> > 
> > On Sun, Apr 24, 2016 at 8:59 PM, Brian Gesiak \<modocache@gmail.com \<mailto:modocache@gmail.com\>\> wrote:  
> > Thanks to this commit \<[https://github.com/apple/swift/commit/ad269b0e1fbc12037ae2c16634b5b451061657c6&gt](https://github.com/apple/swift/commit/ad269b0e1fbc12037ae2c16634b5b451061657c6&gt); it looks as if IsTestCandidate has been moved out of SourceKit and into libIndex:
> > 
> > isTestCandidate(swift::ValueDecl) \<[https://github.com/apple/swift/blob/8dad7f780347788f6032ad9e25ce5340aecf4073/lib/Index/Index.cpp#L754&gt](https://github.com/apple/swift/blob/8dad7f780347788f6032ad9e25ce5340aecf4073/lib/Index/Index.cpp#L754&gt);  
> > swift::index::FuncDeclIndexSymbol.IsTestCandidate \<[https://github.com/apple/swift/blob/41e4e9b6efc745f04df23bd6a803a467c57a66b8/include/swift/Index/IndexSymbol.h#L102&gt](https://github.com/apple/swift/blob/41e4e9b6efc745f04df23bd6a803a467c57a66b8/include/swift/Index/IndexSymbol.h#L102&gt); and where it’s set \<[https://github.com/apple/swift/blob/8dad7f780347788f6032ad9e25ce5340aecf4073/lib/Index/Index.cpp#L786&gt;\](https://github.com/apple/swift/blob/8dad7f780347788f6032ad9e25ce5340aecf4073/lib/Index/Index.cpp#L786&gt;%5C).  
> > I’m looking into adding an option to swiftc to emit XCTest candidate methods. How does swiftc -frontend -dump-xctest-methods sound?
> > 
> > - Brian Gesiak
> > 
> > On Sun, Apr 17, 2016 at 5:50 PM, Drew Crawford \<drew@sealedabstract.com \<mailto:drew@sealedabstract.com\>\> wrote:
> > 
> > > On Apr 3, 2016, at 10:05 PM, Dmitri Gribenko \<gribozavr@gmail.com \<mailto:gribozavr@gmail.com\>\> wrote:
> > > 
> > > > Hmm... but then wouldn't that more tightly couple the test discovery tool  
> > > > and the Swift compiler? In an earlier email you said you "like #3 better  
> > > > [...] because that would decouple the test discovery tool from the Swift  
> > > > compiler." I think that part is confusing me.
> > > 
> > > Sorry -- what I meant is that the compiler remains the point of truth  
> > > about the language and can find the tests. The tools that actually  
> > > generate glue code won't need to parse code, and would be decoupled  
> > > from the compiler.
> > 
> > I am internally shipping a test framework that discovers tests via an out-of-tree parser. Teaching swiftc about XCTest syntax is not sufficient to deprecate my parser, and therefore is not sufficient for the compiler to be the source of truth for my tests.

---

_[View the full topic](https://forums.swift.org/t/sr-710-rfc-automatically-detecting-xctest-test-methods-on-linux-reflection-sourcekit/2050)._