[SR-710][RFC] Automatically detecting XCTest test methods on Linux: Reflection? SourceKit?

Hello all!

SR-710 <Issues · apple/swift-issues · GitHub; tracks a major goal for Swift
3: having SwiftPM/corelibs-xctest automatically generate a list of test
methods to execute. The implementations we’re considering span various
parts of the codebase: libIDE, SourceKit, the reflection APIs, etc. We need
input from people familiar with these components.

Here’s the issue: currently, users of corelibs-xctest must manually list
each test they wish to execute:

class MyTestCase: XCTestCase {
    static var allTests: [(String, MyTestCase -> () throws -> Void)] {
        return [
            ("testFoo", testFoo)
        ]
    }

    func testFoo() {
        XCTAssert(true)
    }

    // The user forgot to list this method in `allTests`, so it is never run.
    func testBar() {
        XCTAssert(true)
    }
}

This is tedious and error-prone. We need to do better by Swift 3!

Apple XCTest uses Objective-C reflection at runtime to compile a list of
NSInvocation to execute as tests. We can’t use the same approach in Swift:
as far as I know, there’s no reflection API that allows us to find instance
methods defined on a class, and adding such an API is considered a stretch
goal for Swift 3
<https://lists.swift.org/pipermail/swift-evolution/Week-of-Mon-20151207/001682.html&gt;
.

Several commenters in SR-710 <Issues · apple/swift-issues · GitHub; have
suggested using libIDE or SourceKit to determine the list of tests. I
believe this is the most feasible approach that could be implemented in
time for Swift 3. A few caveats:

   - SourceKit is coupled to XPC, which only works on OS X.
   - Logic like SourceKit::FuncDeclEntityInfo.IsTestCandidate
   <swift/LangSupport.h at 12593eff135ab2fa99529e9fa8ecc61ce268cd45 · apple/swift · GitHub;
   exists in SourceKit, but not libIDE.
   - libIDE defines a C++ interface, so we could not use it from Swift
   (should we choose to use Swift to generate the list of test methods).

I think we have three concrete options for implementing SR-710
<Issues · apple/swift-issues · GitHub

   1. Port SourceKit to Linux, using a different form of IPC since XPC is
   not available.
   2. Move business logic like IsTestCandidate to libIDE. If the tool to
   generate the list of tests will be written in Swift, we can add a C header
   (like libclang or sourcekitd). The tool would link against libIDE and use
   the C header.
   3. Move business logic like IsTestCandidate to libIDE and add a swiftc
   option to interface with its functionality. The tool that generates the
   list of tests would invoke swiftc to get the list.

I think #2 is the best option. It’s less work than both #1 and #3. I
believe logic like IsTestCandidate belongs in libIDE anyway—SourceKit
should stick to XPC and asynchronous communication with libIDE.

Not being an expert in many of these components, I have several questions:

   - I’m assuming the reflection API to return a list of instance methods
   on a XCTestCase subclass is not ready yet, and won’t be for some time.
   Is this accurate?
   - I’m assuming that SourceKit is intended to be an asynchronous wrapper
   over libIDE, and that logic like IsTestCandidate should be moved to
   libIDE. Is this accurate?
   - I’m assuming that SourceKit is coupled with XPC, and that it would be
   more work to port it to Linux than it would be to move its logic to libIDE.
   Is this accurate?

If you have thoughts/feedback, please reply to this email or comment on
SR-710 <https://bugs.swift.org/browse/SR-710&gt;\. Your input would be greatly
appreciated!!

Brian Gesiak

I think #2 is the best option. It’s less work than both #1 and #3. I believe
logic like IsTestCandidate belongs in libIDE anyway—SourceKit should stick
to XPC and asynchronous communication with libIDE.

I like #3 better (an option to swiftc), because that would decouple
the test discovery tool from the Swift compiler. That would allow you
to use the discovery tool with different compilers. And, because we
would avoid statically linking libIDE, it would mean one less copy of
LLVM, Clang and Swift in the toolchain.

Not being an expert in many of these components, I have several questions:

I’m assuming the reflection API to return a list of instance methods on a
XCTestCase subclass is not ready yet, and won’t be for some time. Is this
accurate?

I think so.

I’m assuming that SourceKit is intended to be an asynchronous wrapper over
libIDE, and that logic like IsTestCandidate should be moved to libIDE. Is
this accurate?

SourceKit has a lot of functionality of its own, but moving this
particular piece of logic to libIDE sounds reasonable.

I’m assuming that SourceKit is coupled with XPC, and that it would be more
work to port it to Linux than it would be to move its logic to libIDE. Is
this accurate?

It is not tightly coupled with XPC, there is a portability layer that
you could implement for Linux. You would need to decide on an IPC
mechanism and serialization format though.

If you have thoughts/feedback, please reply to this email or comment on
SR-710. Your input would be greatly appreciated!!

I'm wondering how feasible is it to change the XCTest API to
accommodate better the Swift language that we have, rather than trying
to add custom tooling to make the existing API work. Adding magic
tooling that adds behavior not present in the language seems unnatural
to me.

Compare with StdlibUnittest -- by using an API to build tests we get
the following advantages:

- We completely avoid having the issue of test discovery, executing
the code discovers the tests. No reflection needed!

- We can add attributes to tests (for example, skip, xfail). In the
current XCTest API this would require adding some kind of user-defined
attributes, which is another language which is a long way from being
designed and implemented.

- We can define data-parameterized tests.

- Tests can be dynamically synthesized by control flow. In the
current XCTest API, dynamically generating tests would mean
dynamically generating methods, which is even more far off than
read-only method reflection.

Dmitri

···

On Sun, Apr 3, 2016 at 2:11 PM, Brian Gesiak <modocache@gmail.com> wrote:

--
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr@gmail.com>*/

Thanks for the feedback, Dmitri!

I think I misstated options #2 and #3: where I typed "move business logic
like IsTestCandidate to libIDE", I meant "move a lot of the
non-XCTest-specific logic from SourceKit to libIDE, such that implementing
IsTestCandidate would be trivial in SourceKit and our test generation
tool". The last thing I'd want to do is couple libIDE and XCTest!

> I think #2 is the best option. It’s less work than both #1 and #3. I
believe
> logic like IsTestCandidate belongs in libIDE anyway—SourceKit should
stick
> to XPC and asynchronous communication with libIDE.

I like #3 better (an option to swiftc), because that would decouple
the test discovery tool from the Swift compiler. That would allow you
to use the discovery tool with different compilers. And, because we
would avoid statically linking libIDE, it would mean one less copy of
LLVM, Clang and Swift in the toolchain.

My concern with adding a swiftc option would be deciding upon an interface
to it. Would we want something like clang-query--a generic method of
finding methods that match a set of criteria? That seems like it would be a
lot of work to implement.

Perhaps I'm misunderstanding what you have in mind for swiftc. Could you
elaborate?

> Not being an expert in many of these components, I have several
questions:
>
> I’m assuming the reflection API to return a list of instance methods on a
> XCTestCase subclass is not ready yet, and won’t be for some time. Is this
> accurate?

I think so.

> I’m assuming that SourceKit is intended to be an asynchronous wrapper
over
> libIDE, and that logic like IsTestCandidate should be moved to libIDE. Is
> this accurate?

SourceKit has a lot of functionality of its own, but moving this
particular piece of logic to libIDE sounds reasonable.

> I’m assuming that SourceKit is coupled with XPC, and that it would be
more
> work to port it to Linux than it would be to move its logic to libIDE. Is
> this accurate?

It is not tightly coupled with XPC, there is a portability layer that
you could implement for Linux. You would need to decide on an IPC
mechanism and serialization format though.

> If you have thoughts/feedback, please reply to this email or comment on
> SR-710. Your input would be greatly appreciated!!

I'm wondering how feasible is it to change the XCTest API to
accommodate better the Swift language that we have, rather than trying
to add custom tooling to make the existing API work. Adding magic
tooling that adds behavior not present in the language seems unnatural
to me.

Compare with StdlibUnittest -- by using an API to build tests we get
the following advantages:

- We completely avoid having the issue of test discovery, executing
the code discovers the tests. No reflection needed!

- We can add attributes to tests (for example, skip, xfail). In the
current XCTest API this would require adding some kind of user-defined
attributes, which is another language which is a long way from being
designed and implemented.

- We can define data-parameterized tests.

- Tests can be dynamically synthesized by control flow. In the
current XCTest API, dynamically generating tests would mean
dynamically generating methods, which is even more far off than
read-only method reflection.

I definitely agree that all of these are fantastic features. However, the
corelibs-xctest README spells out the following goal for the project: "This
version of XCTest uses the same API as the XCTest you are familiar with
from Xcode. Our goal is to enable your project's tests to run on all Swift
platforms without having to rewrite them."

This goal often conflicts with what would be the best API for a Swift
testing framework--we are forced to use mutable reference types, extensive
subclassing, include a ton of optional properties--all in order to mirror
exactly the Apple XCTest API.

I suppose we *could* modify the SDK overlay for XCTest to provide an API
more suitable for Swift, but I think that would involve a lot of
coordination with Apple XCTest and Xcode teams, some swift-evolution
proposals, and a potentially harsh migration path for developers writing
their tests in Apple XCTest. So while I think corelibs-xctest and Apple
XCTest can (and should) evolve to better take advantage of Swift features
over time, I don't think we can migrate both before Swift 3's release.

(+CC Mike Ferris, who works on Apple XCTest and corelibs-xctest.)

- Brian Gesiak

···

On Sun, Apr 3, 2016 at 6:36 PM, Dmitri Gribenko <gribozavr@gmail.com> wrote:

On Sun, Apr 3, 2016 at 2:11 PM, Brian Gesiak <modocache@gmail.com> wrote:

I think #2 is the best option. It’s less work than both #1 and #3. I believe
logic like IsTestCandidate belongs in libIDE anyway—SourceKit should stick
to XPC and asynchronous communication with libIDE.

I like #3 better (an option to swiftc), because that would decouple
the test discovery tool from the Swift compiler. That would allow you
to use the discovery tool with different compilers. And, because we
would avoid statically linking libIDE, it would mean one less copy of
LLVM, Clang and Swift in the toolchain.

Ultimately my opinion is that it is likely that the package manager will want an API interface to Swift in any case. I personally would rather we simply plan on that.

I also would like to avoid duplicating anything in the toolchain, but think that should be done by moving the driver to sitting on top of a shared library.

Not being an expert in many of these components, I have several questions:

I’m assuming the reflection API to return a list of instance methods on a
XCTestCase subclass is not ready yet, and won’t be for some time. Is this
accurate?

I think so.

I’m assuming that SourceKit is intended to be an asynchronous wrapper over
libIDE, and that logic like IsTestCandidate should be moved to libIDE. Is
this accurate?

SourceKit has a lot of functionality of its own, but moving this
particular piece of logic to libIDE sounds reasonable.

I’m assuming that SourceKit is coupled with XPC, and that it would be more
work to port it to Linux than it would be to move its logic to libIDE. Is
this accurate?

It is not tightly coupled with XPC, there is a portability layer that
you could implement for Linux. You would need to decide on an IPC
mechanism and serialization format though.

For our purposes, I don't think we need IPC. I think a direct (C) library interface would be fine. Clients can implement the IPC/XPC if they need it.

If you have thoughts/feedback, please reply to this email or comment on
SR-710. Your input would be greatly appreciated!!

I'm wondering how feasible is it to change the XCTest API to
accommodate better the Swift language that we have, rather than trying
to add custom tooling to make the existing API work. Adding magic
tooling that adds behavior not present in the language seems unnatural
to me.

I agree with you that it is unnatural, but I think this ship has sailed for XCTest, we have a need to support the existing API in a cross platform manner.

My personal preference is that eventually we would build features like this on top of general support for attributes (a la Java/Python/C#).

Compare with StdlibUnittest -- by using an API to build tests we get
the following advantages:

- We completely avoid having the issue of test discovery, executing
the code discovers the tests. No reflection needed!

While I think StdlibUnittest is neat, I also believe that there are very good reasons for supporting test discovery in a test suite. I have used these features in other suites to great avail to create (bidirectional, sometimes) lit bridge adaptors between various test frameworks (Python unittest, googletest, XCTest).

In an IDE context, it is also very useful to be able to perform test discovery independent of test execution.

As one other example, I've used lit with suites with hundreds of thousands of tests... it would be unfortunate to have to dynamically discover all of those tests when the user is just trying to run a single one.

- We can add attributes to tests (for example, skip, xfail). In the
current XCTest API this would require adding some kind of user-defined
attributes, which is another language which is a long way from being
designed and implemented.

This isn't necessarily the case, XCTest could in theory provide explicit APIs to do these things as part of test execution. I agree attributes are a better fit in the current model.

- We can define data-parameterized tests.

- Tests can be dynamically synthesized by control flow. In the
current XCTest API, dynamically generating tests would mean
dynamically generating methods, which is even more far off than
read-only method reflection.

FWIW, XCTest has some support for doing these kinds of things, they just don't take the form of the pattern-matched methods.

- Daniel

···

On Apr 3, 2016, at 3:36 PM, Dmitri Gribenko <gribozavr@gmail.com> wrote:
On Sun, Apr 3, 2016 at 2:11 PM, Brian Gesiak <modocache@gmail.com> wrote:

Dmitri

--
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr@gmail.com>*/

I imagined it would be an XCTest-specific flag, to only find
XCTest-style test methods.

Dmitri

···

On Sun, Apr 3, 2016 at 4:53 PM, Brian Gesiak <modocache@gmail.com> wrote:

My concern with adding a swiftc option would be deciding upon an interface
to it. Would we want something like clang-query--a generic method of finding
methods that match a set of criteria? That seems like it would be a lot of
work to implement.

--
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr@gmail.com>*/

Hmm... but then wouldn't that more tightly couple the test discovery tool
and the Swift compiler? In an earlier email you said you "like #3 better
[...] because that would decouple the test discovery tool from the Swift
compiler." I think that part is confusing me.

In any case, so far it sounds like we'll be moving logic from SourceKit
into libIDE (although we're still mulling over options #2 or #3). Of
course, if anyone recommends a different approach, please reply to this
thread!

- Brian Gesiak

In case anyone interested missed it, John Holdsworth mentions another
approach in a comment on Issues · apple/swift-issues · GitHub using
runtime metadata to find a list of tests. The linked project contains an
example. It:

1. Specifies a test method regex:

2. In Swift, the project exposes metadata information for Swift objects:

3. Uses an UnsafeMutablePointer<ClassMetadataSwift> to get a reference to
the methods defined on an instance:

I assume this approach won't be acceptable to some because it relies on
private information about Swift objects, but I don't know for sure. Please
chime in if this approach seems reasonable to you!

Based on the assumption that we can't use metadata, however, I'm going to
try to begin working on approaches #2 or #3, which I mentioned in my
original email. I think the preliminary work is the same (like
moving SourceKit::FuncDeclEntityInfo.IsTestCandidate to libIDE), so I'll
hold off on making a hard decision between #2 and #3 until necessary.

- Brian Gesiak

···

On Mon, Apr 4, 2016 at 12:23 AM, Daniel Dunbar <daniel_dunbar@apple.com> wrote:

> On Apr 3, 2016, at 3:36 PM, Dmitri Gribenko <gribozavr@gmail.com> wrote:
>
> On Sun, Apr 3, 2016 at 2:11 PM, Brian Gesiak <modocache@gmail.com> > wrote:
>> I think #2 is the best option. It’s less work than both #1 and #3. I
believe
>> logic like IsTestCandidate belongs in libIDE anyway—SourceKit should
stick
>> to XPC and asynchronous communication with libIDE.
>
> I like #3 better (an option to swiftc), because that would decouple
> the test discovery tool from the Swift compiler. That would allow you
> to use the discovery tool with different compilers. And, because we
> would avoid statically linking libIDE, it would mean one less copy of
> LLVM, Clang and Swift in the toolchain.

Ultimately my opinion is that it is likely that the package manager will
want an API interface to Swift in any case. I personally would rather we
simply plan on that.

I also would like to avoid duplicating anything in the toolchain, but
think that should be done by moving the driver to sitting on top of a
shared library.

>> Not being an expert in many of these components, I have several
questions:
>>
>> I’m assuming the reflection API to return a list of instance methods on
a
>> XCTestCase subclass is not ready yet, and won’t be for some time. Is
this
>> accurate?
>
> I think so.
>
>> I’m assuming that SourceKit is intended to be an asynchronous wrapper
over
>> libIDE, and that logic like IsTestCandidate should be moved to libIDE.
Is
>> this accurate?
>
> SourceKit has a lot of functionality of its own, but moving this
> particular piece of logic to libIDE sounds reasonable.
>
>> I’m assuming that SourceKit is coupled with XPC, and that it would be
more
>> work to port it to Linux than it would be to move its logic to libIDE.
Is
>> this accurate?
>
> It is not tightly coupled with XPC, there is a portability layer that
> you could implement for Linux. You would need to decide on an IPC
> mechanism and serialization format though.

For our purposes, I don't think we need IPC. I think a direct (C) library
interface would be fine. Clients can implement the IPC/XPC if they need it.

>> If you have thoughts/feedback, please reply to this email or comment on
>> SR-710. Your input would be greatly appreciated!!
>
> I'm wondering how feasible is it to change the XCTest API to
> accommodate better the Swift language that we have, rather than trying
> to add custom tooling to make the existing API work. Adding magic
> tooling that adds behavior not present in the language seems unnatural
> to me.

I agree with you that it is unnatural, but I think this ship has sailed
for XCTest, we have a need to support the existing API in a cross platform
manner.

My personal preference is that eventually we would build features like
this on top of general support for attributes (a la Java/Python/C#).

> Compare with StdlibUnittest -- by using an API to build tests we get
> the following advantages:
>
> - We completely avoid having the issue of test discovery, executing
> the code discovers the tests. No reflection needed!

While I think StdlibUnittest is neat, I also believe that there are very
good reasons for supporting test discovery in a test suite. I have used
these features in other suites to great avail to create (bidirectional,
sometimes) lit bridge adaptors between various test frameworks (Python
unittest, googletest, XCTest).

In an IDE context, it is also very useful to be able to perform test
discovery independent of test execution.

As one other example, I've used lit with suites with hundreds of thousands
of tests... it would be unfortunate to have to dynamically discover all of
those tests when the user is just trying to run a single one.

> - We can add attributes to tests (for example, skip, xfail). In the
> current XCTest API this would require adding some kind of user-defined
> attributes, which is another language which is a long way from being
> designed and implemented.

This isn't necessarily the case, XCTest could in theory provide explicit
APIs to do these things as part of test execution. I agree attributes are a
better fit in the current model.

> - We can define data-parameterized tests.

> - Tests can be dynamically synthesized by control flow. In the
> current XCTest API, dynamically generating tests would mean
> dynamically generating methods, which is even more far off than
> read-only method reflection.

FWIW, XCTest has some support for doing these kinds of things, they just
don't take the form of the pattern-matched methods.

- Daniel

>
> Dmitri
>
> --
> main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
> (j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr@gmail.com>*/

Sorry -- what I meant is that the compiler remains the point of truth
about the language and can find the tests. The tools that actually
generate glue code won't need to parse code, and would be decoupled
from the compiler.

Dmitri

···

On Sun, Apr 3, 2016 at 7:56 PM, Brian Gesiak <modocache@gmail.com> wrote:

Hmm... but then wouldn't that more tightly couple the test discovery tool
and the Swift compiler? In an earlier email you said you "like #3 better
[...] because that would decouple the test discovery tool from the Swift
compiler." I think that part is confusing me.

--
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr@gmail.com>*/

I am internally shipping a test framework that discovers tests via an out-of-tree parser. Teaching swiftc about XCTest syntax is not sufficient to deprecate my parser, and therefore is not sufficient for the compiler to be the source of truth for my tests.

···

On Apr 3, 2016, at 10:05 PM, Dmitri Gribenko <gribozavr@gmail.com> wrote:

Hmm... but then wouldn't that more tightly couple the test discovery tool
and the Swift compiler? In an earlier email you said you "like #3 better
[...] because that would decouple the test discovery tool from the Swift
compiler." I think that part is confusing me.

Sorry -- what I meant is that the compiler remains the point of truth
about the language and can find the tests. The tools that actually
generate glue code won't need to parse code, and would be decoupled
from the compiler.

Thanks to this commit
<https://github.com/apple/swift/commit/ad269b0e1fbc12037ae2c16634b5b451061657c6&gt;
it looks as if IsTestCandidate has been moved out of SourceKit and into
libIndex:

   - isTestCandidate(swift::ValueDecl)
   <https://github.com/apple/swift/blob/8dad7f780347788f6032ad9e25ce5340aecf4073/lib/Index/Index.cpp#L754&gt;
   - swift::index::FuncDeclIndexSymbol.IsTestCandidate
   <https://github.com/apple/swift/blob/41e4e9b6efc745f04df23bd6a803a467c57a66b8/include/swift/Index/IndexSymbol.h#L102&gt;
   and where it’s set
   <https://github.com/apple/swift/blob/8dad7f780347788f6032ad9e25ce5340aecf4073/lib/Index/Index.cpp#L786&gt;
   .

I’m looking into adding an option to swiftc to emit XCTest candidate
methods. How does swiftc -frontend -dump-xctest-methods sound?

- Brian Gesiak

···

On Sun, Apr 17, 2016 at 5:50 PM, Drew Crawford <drew@sealedabstract.com> wrote:

On Apr 3, 2016, at 10:05 PM, Dmitri Gribenko <gribozavr@gmail.com> wrote:

Hmm... but then wouldn't that more tightly couple the test discovery tool
and the Swift compiler? In an earlier email you said you "like #3 better
[...] because that would decouple the test discovery tool from the Swift
compiler." I think that part is confusing me.

Sorry -- what I meant is that the compiler remains the point of truth
about the language and can find the tests. The tools that actually
generate glue code won't need to parse code, and would be decoupled
from the compiler.

I am internally shipping a test framework that discovers tests via an
out-of-tree parser. Teaching swiftc about XCTest syntax is not sufficient
to deprecate my parser, and therefore is not sufficient for the compiler to
be the source of truth for my tests.

I made an attempt at adding `swiftc -frontend -dump-xctest-methods`:

Feedback appreciated!!

- Brian Gesiak

···

On Sun, Apr 24, 2016 at 8:59 PM, Brian Gesiak <modocache@gmail.com> wrote:

Thanks to this commit
<https://github.com/apple/swift/commit/ad269b0e1fbc12037ae2c16634b5b451061657c6&gt;
it looks as if IsTestCandidate has been moved out of SourceKit and into
libIndex:

   - isTestCandidate(swift::ValueDecl)
   <https://github.com/apple/swift/blob/8dad7f780347788f6032ad9e25ce5340aecf4073/lib/Index/Index.cpp#L754&gt;
   - swift::index::FuncDeclIndexSymbol.IsTestCandidate
   <https://github.com/apple/swift/blob/41e4e9b6efc745f04df23bd6a803a467c57a66b8/include/swift/Index/IndexSymbol.h#L102&gt;
   and where it’s set
   <https://github.com/apple/swift/blob/8dad7f780347788f6032ad9e25ce5340aecf4073/lib/Index/Index.cpp#L786&gt;
   .

I’m looking into adding an option to swiftc to emit XCTest candidate
methods. How does swiftc -frontend -dump-xctest-methods sound?

- Brian Gesiak

On Sun, Apr 17, 2016 at 5:50 PM, Drew Crawford <drew@sealedabstract.com> > wrote:

On Apr 3, 2016, at 10:05 PM, Dmitri Gribenko <gribozavr@gmail.com> wrote:

Hmm... but then wouldn't that more tightly couple the test discovery tool
and the Swift compiler? In an earlier email you said you "like #3 better
[...] because that would decouple the test discovery tool from the Swift
compiler." I think that part is confusing me.

Sorry -- what I meant is that the compiler remains the point of truth
about the language and can find the tests. The tools that actually
generate glue code won't need to parse code, and would be decoupled
from the compiler.

I am internally shipping a test framework that discovers tests via an
out-of-tree parser. Teaching swiftc about XCTest syntax is not sufficient
to deprecate my parser, and therefore is not sufficient for the compiler to
be the source of truth for my tests.

Sorry for the delay in following up.

I have had several long discussions on this topic with Dmitri and others, which
I will try to summarize here:

*TL;DR*: We think the right long-term path forward is to pursue porting
SourceKit to Linux, and would like to explore that direction first before trying
to develop a compromise which either (a) uses internal and likely-to-break APIs
(e.g., dump-ast) or (b) devotes significant engineering resources to a "right
solution" for this problem, but which will only solve this problem and not other
issues where having an API for use with the Swift language.

## Problem Statement

The XCTest API has historically been defined in terms of methods with a
particular naming convention `test...` which were in subclasses of XCTestCase.

On OS X, these methods can be found via the Objective-C runtime but that does
not work on Linux. Our current solution on Linux requires manual specification
of all of the test methods, and is a huge maintenance burden for people trying
to use XCTest on Linux or maintain cross-platform projects.

## Background

This feature works on OS X in two ways:

1. As mentioned, tests are run by dynamic discovery through the Objective-C
   runtime when the bundle containing the tests is executed.
   
2. Within the Xcode IDE, tests are "discovered" (for use in UI) through the use
   of the Xcode indexer.
   
   The mechanism at work here, for Swift, is a combination of functionality in
   the indexing engine (to aggregate and query results) and the raw underlying
   data provided by SourceKit.
   
   We would like any implementation of this feature to share as much code as
   possible with SourceKit and Xcode's implementation in order for
   cross-platform projects to behave predictably.
   
It also happens that SwiftPM has several other needs for API-based interactions
with the functionality in the Swift compiler. Several examples:
   
* We would like to be able to enforce the strictness of the `Package.swift`
  manifest file format. This requires APIs to interact with the Swift
  AST. [SR-1433] [SwiftPM] Enforce manifest strictness · Issue #5293 · apple/swift-package-manager · GitHub
  
* We would like to support advanced features for dealing with automatic
  inter-module dependencies. For example, one idea which has been proposed is
  that if a module attempted to import a module upon which it did not have a
  dependency, that we would prompt the developer if this dependency should
  automatically be added to the manifest. Doing this feature well requires
  APIs to interact with the Swift compiler as it parses source code and
  reports diagnostics.
  
* We would like to understand the current level of parallelism being used by
  the Swift compiler, so that llbuild can accurately schedule work. This
  requires APIs for interacting with an in-flight compilation process.

## Discussion

We discussed several avenues of attack on this problem. I will go through them
one by one. I am just attempting to summarize the conversation, obviously there
is a ton of nuance to each point, and hopefully other people will chime in if I
missed something important.

### Language Features

One way to view this problem is that XCTest's API (i.e., depending on test
naming and dynamic discovery) is a poor fit for Swift today, and that this
problem should be tackled in that direction.

For example, the Swift compiler itself has a test framework that does not depend
on dynamic discovery. One could also imagine language-level features which would
solve the arbitrary problem of wanting to find discoverable things; that could
take the form of an `@XCTest` attribute, or "generalized attribute" support.

The current mission for Swift 3, however, is API parity between Linux and OS X,
and so this direction does not lead to any short term solution. For that reason,
while many of us ultimately think this is the right long term direction, we need
to find another one as well.

### Custom "Supported" XCTest Feature

The next option is to pursue a custom, but "supported" feature intended to
tackle this problem head on. Some proposals that have come up are, e.g., a new
compiler flag which emits the list of test methods.

This approach has a couple unfortunate properties:

1. It is non-trivial. We can either design this as an incredibly XCTest specific
   feature requiring understanding of the backend (compiler directly emits
   metadata into .o file), or a midway feature (compiler tells us list of test
   methods, but then we have to manage incremental compilation and the desire to
   not compile things multiple times more than necessary ourselves).
   
2. It is not-reusable. The work we do here doesn't help with any of the other
   ways we want to use the Swift compiler as an API.
   
### Implement an API-based Interface

This approach means porting SourceKit to Linux, and then building this feature
on top of that (that itself will be non-trivial, which is something not to be
glossed over).

Everyone generally agrees we should have some kind of API-based access to the
compiler, so this is in line with our long-term direction, but in a way which is
actionable now (it does not require design).

This approach has its own issues:

1. Porting SourceKit may require a fair amount of work. The current code base is
   very OS X specific in particular areas (generally around the use of
   `libxpc`). This will require adding abstractions, developing alternatives,
   and writing the code and build system changes to get this to happen.
   
2. Somehwat unrelated, but the compiler itself (`swiftc`) is not yet written in
   a way that it can be used from SourceKit. If we do port and use SourceKit, it
   will mean the toolchains increase in size because we effectively will have
   two copies of many parts of Swift/Clang/LLVM installed.
   
3. By itself, this does not fully solve the issue. As mentioned in the
   background, SourceKit itself does not directly manage all parts of XCTest
   discovery today on OS X, it is a collaboration between SourceKit and the
   Xcode indexer to do discovery without execution. The expectation, however, is
   that we can, however, use SourceKit's APIs to implement this feature in a way
   that is "supported".

## Conclusion

The conclusion was that after weighing all of the tradeoffs, it made the most
sense to encourage porting of SourceKit to Linux and then using it to build out
the Linux test discovery feature. This was most in line with a desirable
long-term direction without being blocked on language design.

I don't think we are particularly firm in this conclusion. If you feel that the work
you have already invested gets us so close that it is worth prioritizing that
approach, then please push back.

- Daniel

···

On May 1, 2016, at 8:19 PM, Brian Gesiak <modocache@gmail.com> wrote:

I made an attempt at adding `swiftc -frontend -dump-xctest-methods`: [SR-710][Frontend] Add `-dump-xctest-methods` by modocache · Pull Request #2364 · apple/swift · GitHub

Feedback appreciated!!

- Brian Gesiak

On Sun, Apr 24, 2016 at 8:59 PM, Brian Gesiak <modocache@gmail.com <mailto:modocache@gmail.com>> wrote:
Thanks to this commit <https://github.com/apple/swift/commit/ad269b0e1fbc12037ae2c16634b5b451061657c6&gt; it looks as if IsTestCandidate has been moved out of SourceKit and into libIndex:

isTestCandidate(swift::ValueDecl) <https://github.com/apple/swift/blob/8dad7f780347788f6032ad9e25ce5340aecf4073/lib/Index/Index.cpp#L754&gt;
swift::index::FuncDeclIndexSymbol.IsTestCandidate <https://github.com/apple/swift/blob/41e4e9b6efc745f04df23bd6a803a467c57a66b8/include/swift/Index/IndexSymbol.h#L102&gt; and where it’s set <https://github.com/apple/swift/blob/8dad7f780347788f6032ad9e25ce5340aecf4073/lib/Index/Index.cpp#L786&gt;\.
I’m looking into adding an option to swiftc to emit XCTest candidate methods. How does swiftc -frontend -dump-xctest-methods sound?

- Brian Gesiak

On Sun, Apr 17, 2016 at 5:50 PM, Drew Crawford <drew@sealedabstract.com <mailto:drew@sealedabstract.com>> wrote:

On Apr 3, 2016, at 10:05 PM, Dmitri Gribenko <gribozavr@gmail.com <mailto:gribozavr@gmail.com>> wrote:

Hmm... but then wouldn't that more tightly couple the test discovery tool
and the Swift compiler? In an earlier email you said you "like #3 better
[...] because that would decouple the test discovery tool from the Swift
compiler." I think that part is confusing me.

Sorry -- what I meant is that the compiler remains the point of truth
about the language and can find the tests. The tools that actually
generate glue code won't need to parse code, and would be decoupled
from the compiler.

I am internally shipping a test framework that discovers tests via an out-of-tree parser. Teaching swiftc about XCTest syntax is not sufficient to deprecate my parser, and therefore is not sufficient for the compiler to be the source of truth for my tests.

For whatever it's worth, this direction is a win on my side as well.

In addition to the problem of test discovery (for which I'm using an out-of-tree parser), I have a lot of other problems entirely outside of testing that rely on source-level queries similar to the XCTest problem. This is things like parsing comments for documentation, implementing dispatch-by-string, etc. I currently rely on SK in many cases, but lack of support on Linux is a major issue. Lack of features exposed in the SK APIs is another issue.

IMO it is a clear win to invest in resolving these problems inside SK. Right now it is basically a glorified Xcode daemon, but I think it can have a bright future as a multi-client tool if we're willing to invest in making that happen.

···

On May 6, 2016, at 3:04 PM, Daniel Dunbar <daniel_dunbar@apple.com> wrote:

The conclusion was that after weighing all of the tradeoffs, it made the most
sense to encourage porting of SourceKit to Linux and then using it to build out
the Linux test discovery feature. This was most in line with a desirable
long-term direction without being blocked on language design.

Thanks for the feedback, everyone!

Porting SourceKit to Linux seems like a reasonable solution to me. Still,
there are 354 lines of code in tools/SourceKit that reference "XPC", so a
Linux port will take more than a few lines of source code changes.

I imagine we'll need to insert some sort of shim layer that will use libxpc
on OS X, and a hand-rolled solution for Linux. Alternatively, if anyone
knows of a good open-source library that implements IPC for Linux (and that
has a permissible license), that would be a great help here.

I've also seen the idea proposed that Apple could open-source libxpc, which
we could then port to Linux. This would involve less work than installing a
shim layer in SourceKit, then in addition implementing a Linux IPC library
behind the shim. I don't know who I could talk about making this happen,
but in any case, I filed a Radar:

* rdar://26187442
* rdar://26187442: libxpc should be open-sourced

2. Somehwat unrelated, but the compiler itself (`swiftc`) is not yet

written in a way that it can be used from SourceKit.

Could you explain this further?

- Brian Gesiak

···

On Sat, May 7, 2016 at 4:18 AM, Drew Crawford <drew@sealedabstract.com> wrote:

On May 6, 2016, at 3:04 PM, Daniel Dunbar <daniel_dunbar@apple.com> wrote:

The conclusion was that after weighing all of the tradeoffs, it made the
most
sense to encourage porting of SourceKit to Linux and then using it to
build out
the Linux test discovery feature. This was most in line with a desirable
long-term direction without being blocked on language design.

For whatever it's worth, this direction is a win on my side as well.

In addition to the problem of test discovery (for which I'm using an
out-of-tree parser), I have a lot of other problems entirely outside of
testing that rely on source-level queries similar to the XCTest problem.
This is things like parsing comments for documentation, implementing
dispatch-by-string, etc. I currently rely on SK in many cases, but lack of
support on Linux is a major issue. Lack of features exposed in the SK APIs
is another issue.

IMO it is a clear win to invest in resolving these problems inside SK.
Right now it is basically a glorified Xcode daemon, but I think it can have
a bright future as a multi-client tool if we're willing to invest in making
that happen.

SourceKit actually supports "pluggable" transports. There are two:
XPC, and an in-memory one (which could be bit-rotten a bit). You can
either add the third one, or try to bootstrap with the in-memory one
for the first pass.

Dmitri

···

On Mon, May 9, 2016 at 7:35 PM, Brian Gesiak <modocache@gmail.com> wrote:

Thanks for the feedback, everyone!

Porting SourceKit to Linux seems like a reasonable solution to me. Still,
there are 354 lines of code in tools/SourceKit that reference "XPC", so a
Linux port will take more than a few lines of source code changes.

I imagine we'll need to insert some sort of shim layer that will use libxpc
on OS X, and a hand-rolled solution for Linux. Alternatively, if anyone
knows of a good open-source library that implements IPC for Linux (and that
has a permissible license), that would be a great help here.

--
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr@gmail.com>*/

Thanks for the feedback, everyone!

Porting SourceKit to Linux seems like a reasonable solution to me. Still, there are 354 lines of code in tools/SourceKit that reference "XPC", so a Linux port will take more than a few lines of source code changes.

I imagine we'll need to insert some sort of shim layer that will use libxpc on OS X, and a hand-rolled solution for Linux. Alternatively, if anyone knows of a good open-source library that implements IPC for Linux (and that has a permissible license), that would be a great help here.

I've also seen the idea proposed that Apple could open-source libxpc, which we could then port to Linux. This would involve less work than installing a shim layer in SourceKit, then in addition implementing a Linux IPC library behind the shim. I don't know who I could talk about making this happen, but in any case, I filed a Radar:

* rdar://26187442
* rdar://26187442: libxpc should be open-sourced
> 2. Somehwat unrelated, but the compiler itself (`swiftc`) is not yet written in a way that it can be used from SourceKit.

Could you explain this further?

Basically, I just meant that SourceKit doesn't currently have APIs for driving the compiler (driver), just interrogating the AST.

- Daniel

···

On May 9, 2016, at 7:35 PM, Brian Gesiak <modocache@gmail.com> wrote:

- Brian Gesiak

On Sat, May 7, 2016 at 4:18 AM, Drew Crawford <drew@sealedabstract.com <mailto:drew@sealedabstract.com>> wrote:

On May 6, 2016, at 3:04 PM, Daniel Dunbar <daniel_dunbar@apple.com <mailto:daniel_dunbar@apple.com>> wrote:

The conclusion was that after weighing all of the tradeoffs, it made the most
sense to encourage porting of SourceKit to Linux and then using it to build out
the Linux test discovery feature. This was most in line with a desirable
long-term direction without being blocked on language design.

For whatever it's worth, this direction is a win on my side as well.

In addition to the problem of test discovery (for which I'm using an out-of-tree parser), I have a lot of other problems entirely outside of testing that rely on source-level queries similar to the XCTest problem. This is things like parsing comments for documentation, implementing dispatch-by-string, etc. I currently rely on SK in many cases, but lack of support on Linux is a major issue. Lack of features exposed in the SK APIs is another issue.

IMO it is a clear win to invest in resolving these problems inside SK. Right now it is basically a glorified Xcode daemon, but I think it can have a bright future as a multi-client tool if we're willing to invest in making that happen.

Thanks for the feedback, everyone!

Porting SourceKit to Linux seems like a reasonable solution to me. Still, there are 354 lines of code in tools/SourceKit that reference "XPC", so a Linux port will take more than a few lines of source code changes.

I imagine we'll need to insert some sort of shim layer that will use libxpc on OS X, and a hand-rolled solution for Linux. Alternatively, if anyone knows of a good open-source library that implements IPC for Linux (and that has a permissible license), that would be a great help here.

Note that implementing IPC for linux is not ‘essential’ to get SourceKit working, we could have it use in-memory C++ dictionaries/arrays objects for the ‘communication’ part.

Also, on the topic of detecting the unit tests via the index request of SourceKit, see my comments in [docs] Add SourceKit's `is_test_candidate` by modocache · Pull Request #2455 · apple/swift · GitHub on an enhancement to assist with this.

···

On May 9, 2016, at 7:35 PM, Brian Gesiak via swift-dev <swift-dev@swift.org> wrote:

I've also seen the idea proposed that Apple could open-source libxpc, which we could then port to Linux. This would involve less work than installing a shim layer in SourceKit, then in addition implementing a Linux IPC library behind the shim. I don't know who I could talk about making this happen, but in any case, I filed a Radar:

* rdar://26187442
* rdar://26187442: libxpc should be open-sourced
> 2. Somehwat unrelated, but the compiler itself (`swiftc`) is not yet written in a way that it can be used from SourceKit.

Could you explain this further?

- Brian Gesiak

On Sat, May 7, 2016 at 4:18 AM, Drew Crawford <drew@sealedabstract.com <mailto:drew@sealedabstract.com>> wrote:

On May 6, 2016, at 3:04 PM, Daniel Dunbar <daniel_dunbar@apple.com <mailto:daniel_dunbar@apple.com>> wrote:

The conclusion was that after weighing all of the tradeoffs, it made the most
sense to encourage porting of SourceKit to Linux and then using it to build out
the Linux test discovery feature. This was most in line with a desirable
long-term direction without being blocked on language design.

For whatever it's worth, this direction is a win on my side as well.

In addition to the problem of test discovery (for which I'm using an out-of-tree parser), I have a lot of other problems entirely outside of testing that rely on source-level queries similar to the XCTest problem. This is things like parsing comments for documentation, implementing dispatch-by-string, etc. I currently rely on SK in many cases, but lack of support on Linux is a major issue. Lack of features exposed in the SK APIs is another issue.

IMO it is a clear win to invest in resolving these problems inside SK. Right now it is basically a glorified Xcode daemon, but I think it can have a bright future as a multi-client tool if we're willing to invest in making that happen.

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

There is libxpc from NextBSD:

You’d still need to implement Mach IPC on top of something though.

— Luke

···

On 10 May 2016, at 12:35 PM, Brian Gesiak via swift-dev <swift-dev@swift.org> wrote:

Thanks for the feedback, everyone!

Porting SourceKit to Linux seems like a reasonable solution to me. Still, there are 354 lines of code in tools/SourceKit that reference "XPC", so a Linux port will take more than a few lines of source code changes.

I imagine we'll need to insert some sort of shim layer that will use libxpc on OS X, and a hand-rolled solution for Linux. Alternatively, if anyone knows of a good open-source library that implements IPC for Linux (and that has a permissible license), that would be a great help here.

I've also seen the idea proposed that Apple could open-source libxpc, which we could then port to Linux. This would involve less work than installing a shim layer in SourceKit, then in addition implementing a Linux IPC library behind the shim. I don't know who I could talk about making this happen, but in any case, I filed a Radar:

* rdar://26187442
* rdar://26187442: libxpc should be open-sourced
> 2. Somehwat unrelated, but the compiler itself (`swiftc`) is not yet written in a way that it can be used from SourceKit.

Could you explain this further?

- Brian Gesiak

Or maybe not, looks like there’s a Unix transport in transports/unix.c.

···

On 10 May 2016, at 1:48 PM, Luke Howard via swift-dev <swift-dev@swift.org> wrote:

There is libxpc from NextBSD:

GitHub - jceel/libxpc: Open-source reimplementation of Apple XPC library.

You’d still need to implement Mach IPC on top of something though.

— Luke

On 10 May 2016, at 12:35 PM, Brian Gesiak via swift-dev <swift-dev@swift.org <mailto:swift-dev@swift.org>> wrote:

Thanks for the feedback, everyone!

Porting SourceKit to Linux seems like a reasonable solution to me. Still, there are 354 lines of code in tools/SourceKit that reference "XPC", so a Linux port will take more than a few lines of source code changes.

I imagine we'll need to insert some sort of shim layer that will use libxpc on OS X, and a hand-rolled solution for Linux. Alternatively, if anyone knows of a good open-source library that implements IPC for Linux (and that has a permissible license), that would be a great help here.

I've also seen the idea proposed that Apple could open-source libxpc, which we could then port to Linux. This would involve less work than installing a shim layer in SourceKit, then in addition implementing a Linux IPC library behind the shim. I don't know who I could talk about making this happen, but in any case, I filed a Radar:

* rdar://26187442 <rdar://26187442>
* rdar://26187442: libxpc should be open-sourced
> 2. Somehwat unrelated, but the compiler itself (`swiftc`) is not yet written in a way that it can be used from SourceKit.

Could you explain this further?

- Brian Gesiak

_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev

--
www.lukehoward.com
soundcloud.com/lukehoward