Modularizing Unit Testing on Linux

Hi,

I am working on creating my own Matrix operations using Swift. I have a code for working Matrix operations and written few Unit test cases in a single file. Now I want to segregate the unit test cases in to different swift files. For example MatrixIntTests.swift and MatrixFloatTests.swift etc Where I would like to segregate the operations done based on data type or based on type of operations done on matrix say for example algebraic operations or statistical operations on the matrix. I'm trying to follow the unit test cases done in [Plot package] (Plot/Tests/PlotTests at master Ā· JohnSundell/Plot Ā· GitHub). Plot unit tests works perfectly on my linux machine. But I get the following error when I try to do similar changes in my Matrix libarary's XCTestManifests.swift
error: use of undeclared type 'Linux' public func allTests() -> [Linux.TestCase]

Any inputs on how to get it corrected will be helpful. Any materials which is a good source to know how XCTests works on Linux will be equally appreciated. Or how the LinuxMain.swift and XCTestManifests.swift interact or the order of execution while running the Unit test cases in Linux environment will be really useful to know.

Best Regards,
Santhosh K S

It looks like the project you're following has defined some infrastructure code for Linux in https://github.com/JohnSundell/Plot/blob/master/Tests/PlotTests/LinuxCompatibility.swift that you may be missing.

You can look at something like example-package-fisheryates/Tests at main Ā· apple/example-package-fisheryates Ā· GitHub to see how this is setup in a stock SwiftPM project.

Another option is automatic test discovery, which we are trying to make the default in the future. A good writeup about this feature is Automatic test discovery in Swift on Linux ā€“ Ole Begemann

Hi @NeoNacho,

Thanks for your response. Sorry for my delayed response. I didn't have internet access for couple of days. I will look in to the links which you have shared. And get back with my questions.

Best Regards,
SantMan

Currently my unit tests are arranged in the same way and able to build the test cases. But, My qustion is not how to get the unit test working on Linux platform. You can check my unittests which works and follows the similar patter you are refering to. My question is more on how to segregate the unit tests. I have only one Matrix.swift file under Sources/Matrix directory. And with the same source structure I would like to create multiple Unit test files for the single Matrix.swift file. Say for example MatrixIntTests.swift and MatrixFloatTests.swift. Both of these Test files needs to import from the same Matrix.swift file.

I have gone through this thread, which again talks about single Main.swift file and single MainTests.swift file and explains how it can be dockerized. Again I want to know how to link the multiple Test swift files in to a single Main swift file under the Source/XYZ directory.

It took some time to understand how it is done in Linux. Got it working as I was expecting. Trick is to work with the XCTestManifests.swift file. Here is the code changes if any one is interested in to get it working. Will write a detailed blog and post the details on how to do it in Linux platforms.

Is there a reason you are manually listing the tests and not just using the test discovery flag?

swift test --enable-test-discovery

Also, I donā€™t see anything public anywhere in your library, so no one will be able to use anything in it. You might want to read about Access Control.

Hi @SDGGiesbrecht,
I did try to use it as @NeoNacho suggested. But I didn't see it generate any code for me. Here is the sample output.

 swift test --enable-test-discovery
Test Suite 'All tests' started at 2020-08-31 08:18:43.084
Test Suite 'debug.xctest' started at 2020-08-31 08:18:43.086
Test Suite 'MatrixOperationsTests' started at 2020-08-31 08:18:43.086
Test Case 'MatrixOperationsTests.testMatrixEquality' started at 2020-08-31 08:18:43.086
Test Case 'MatrixOperationsTests.testMatrixEquality' passed (0.0 seconds)
Test Case 'MatrixOperationsTests.testMatrixGreaterOrEqual' started at 2020-08-31 08:18:43.086
Test Case 'MatrixOperationsTests.testMatrixGreaterOrEqual' passed (0.0 seconds)
Test Case 'MatrixOperationsTests.testMatrixSmallerOrEqual' started at 2020-08-31 08:18:43.086
Test Case 'MatrixOperationsTests.testMatrixSmallerOrEqual' passed (0.0 seconds)
Test Suite 'MatrixOperationsTests' passed at 2020-08-31 08:18:43.086
         Executed 3 tests, with 0 failures (0 unexpected) in 0.0 (0.0) seconds
Test Suite 'debug.xctest' passed at 2020-08-31 08:18:43.087
         Executed 3 tests, with 0 failures (0 unexpected) in 0.0 (0.0) seconds
Test Suite 'All tests' passed at 2020-08-31 08:18:43.087
         Executed 3 tests, with 0 failures (0 unexpected) in 0.0 (0.0) seconds

There are few things I noticed.

  1. It doesn't run all the tests. It ran only One class of test which has 3 tests in it.
  2. It didn't generate any code for me.
  3. When I add a new test case Without updating the XCTestManifests.swift file, it doesn't discover the test case as the option says.
    If I delete/comment the contents of XCTestManifests.swift file as shown below
      public func __allTests() -> [XCTestCaseEntry] {
            return [
               //testCase(MatrixIntAndUintInitTests.__allTests__MatrixInitTests),
               //testCase(MatrixFloatAndDoubleInitTests.__allTests__MatrixInitTests),
               //testCase(MatrixIndexingOperations.__allTests__MatrixIndexTests),
               //testCase(MatrixOperationsTests.__allTests__MatrixOperationsTests),
            ]
        }

I get the following output.
Test Suite 'All tests' started at 2020-08-31 08:28:05.467
Test Suite 'debug.xctest' started at 2020-08-31 08:28:05.470
Test Suite 'All tests' passed at 2020-08-31 08:28:05.471

Again I don't see any files being updated.

I am not sure what is that I am missing here? Does this works only on Mac? Can you please point me the mistake I am doing here? Which files it is suppose to generate? Some info on this will be helpful.

Just to be more clear with my setup. Here are the details about my current setup

  1. I am using a Ubuntu-18.04 on my HP laptop.
  2. Swift version 5.2.4 (swift-5.2.4-RELEASE)
  3. I am using Visual studio code on my laptop and I Have configured the code completion and other things to make my life little easier.
  4. I use CLI to run these commands and also configured VS code to do the same.

Thanks for the inputs. I just started with Swift couple of weeks back. And I was stuck at getting the Unit tests to work and spending time in getting it corrected before I proceed. At the moment I am not sure which API's I want to expose and which one needs to be private. I am trying to mimic the numpy library like syntax and API's. Will correct the same as required.

Itā€™s not supposed to generate any file. The point of it is you donā€™t need a checkedā€in list at all.

That sounds fishy. Iā€™ll fork, clone and toy with your specific package to see whatā€™s going on.

1 Like

Fixed it up with this pull request. Just remember that test method names need to start with test.

1 Like

Thanks a lot for taking time to look in to the issue and help me get it fixed. It is working as expected now without the XCTestManifest.swift file. Can you please point me to the document where it documented to use the name of the test to start with test?

The documentation for XCTest (which is what SwiftPM uses) is hosted here. The paragraph right before the first example in the overview describes how XCTest defines what is and isnā€™t a test.

1 Like