[PITCH] Possible solutions to enable the removal of LinuxMain.swift

Development Package Manager
,
1

Currently, those developing for Linux platforms (or cross platform) must use a combination of __allTests and LinuxMain.swift to be able to execute their tests on Linux.

I attempted to look at a number of solutions at try! Swift SJ yesterday and thought I'd throw it out to the forums for proper input before digging any further since this could be far reaching.

There were a number of options I looked at:

  • Regex! :smiley: - not really a maintainable solution
  • Adding SourceKit as a C based dependency to SPM. This should work relatively well but has a number of problems. First, this puts all the logic in SPM, which means corelibs-foundation can't take advantage of it. Second it adds a new dependency that's currently inside the swift repository. Adding it was a manual process of copying source code into a C based target for SPM, so updates would be manual and slow.
  • Additionally, using SourceKit-LSP is not possible since it has a dependency on SPM.
  • Use Reflection to mimic what Objective-C does - currently Swift's reflection is not mature enough for this to work. One option would be wait until it was, but that could take a long time.
  • Changing XCTest to add an API to ask it for the list of tests.

The last solution seems to me (at least) to be the most sensible that has the largest benefit across the different libraries. Again, this can be achieved in a number of ways:

  • Integrate SourceKit - see above for problems with this (unless there's a better way that I'm not aware of)
  • Use symbol demangling on the test binary to pull out subclasses of XCTestCase and find methods inside of these that start with test. @spevans and I looked at this quickly and it seems to be relatively straightforward. Would need to shell out to a process to call swift demangle but most of the searching can be done with Swift code at least. An API for the demangle command (which I understand has been talked about) would allow this to become all Swift code.
  • Use runtime metadata to find the tests.

A number of these approaches have been covered here but it was a couple of years ago and things may have changed (ABI stability, SourceKit available on Linux etc).

Thoughts?

6 Likes
2

Another option suggested has been integrating https://github.com/apple/swift-syntax

However using something like this (like Regex) would allow you to build your test code, add a new test to the file and run the tests before building again which would then pick up the new (but unbuilt) test, which could cause issues

1 Like
3

I think eventually XCTest should automatically discover tests via reflection but I don't know how far away is that. SourceKit seems like the decent solution and we can resolve the integration issues but it seems a little overkill to run it during a build.

I wonder if we can enhance the compiler to produce a file (via some flag) that contains the list of test methods when compiling test targets. SwiftPM can then take that list and generate the appropriate linux main file during the build. In theory, this should be robust and cheaper than other alternatives but IDK if it's feasible/appropriate to add this enhancement in the compiler.

/cc @jrose, @beccadax, @harlanhaskins

2 Likes
4

FYI, there is a working solution now: Test discovery on Linux

2 Likes