My SwiftPM wishlist (aka proposal proposals)

Maybe we should just add some kind of like SwiftPM API for test reporting, kind of like “extensible test tools” that then can be slotted into?

1 Like

Fuzz testing would be awesome! I recently stumbled across GitHub - loiclec/FuzzCheck: Swift package fuzzing engine. The author has moved on to Rust, but the idea of coverage-guided generators for property-based testing is really cool.

2 Likes

Swift even already has the support: swiftc supports fuzzing. We're just missing the ability of SwiftPM to understand that fuzz targets are compiled weirdly (they build binaries, not libraries, but they don't define their own main function so they need -parse-as-library).

3 Likes

Yeah, this feels like it might be a relatively small proposal / technical RFC to figure out what is left to get this more accessible in SwiftPM.

1 Like

@Aciid and I have kicked this around before: I suggested it seemed like a new target type (fuzzTarget!) but IIRC Ankit had some other ideas.

I wonder, could this be solved by SE-0281's proposed @main attribute?

2 Likes

I would like a way to get functionality of if #available tied into the manifest.

Right now if a dependency has a higher deployment target than the top‐level package, graph resolution fails. Instead it would be nice if resolution would succeed and the module would build and import, but all use sites would need to be gated behind the corresponding if #available check. Then it would work the same as system modules like SwiftUI.

The issue came up again in SwiftPM itself here. @NeoNacho

Would this require a full proposal, or just an implementation? It seems straightforward to me. On the other hand I don’t know if the infrastructure exists yet for marking whole modules @available. So while I’m willing to help, I don’t really know where to start.

Isn't there more to it and we also have to make sure those modules don't get loaded at runtime on lower deployment targets? I believe for system frameworks this works by using weak linking and the fact that they are simply not present on newer OSes, right?

I don’t really know how it works under the hood with system libraries. I haven’t dealt with those sorts of low‐level things much yet.

Binary dependencies will probably be more trouble. But for source dependencies, shouldn’t it be possible to do something like tag every symbol in the module with @available(...), similar to how the cross‐module optimization pass adds @inlinable? Then the actual compile‐time deployment target could be walked back to mach the client. That way the module itself would still be present and safely loadable on older platforms, it would just be devoid of any usable symbols. From there, @available and if #available would interact with each other the same as they do from inside the same module.

Only tangentially related, but it would also be nice to be able to walk all dependency deployment targets forward to match the top level package in order to get better optimization. And the ability to suppress dependency warnings would be icing on the cake.

A few things that could be possibly added to the list:

  1. Renaming LinuxMain.swift to TestMain.swift or something that doesn't explicitly specify the platform. I'm working on XCTest support for WebAssembly, where the use of LinuxMain.swift is required, and I imagine XCTest on Windows and other platforms (BSD etc) will need the main file too. It no longer makes sense to keep it named that way, as the number of platforms using it expands to more than just Linux.
  2. "Remote runners" for cross-compiled tests. This is already implicitly supported in Xcode (but not in SwiftPM) when you run your test suite on an iOS device or a simulator. When cross-compiling to WebAssembly, you always need to pass the final test binary to some kind of runtime, either to wasmer with one of its WASI backends, or to a browser to get access to a proper DOM environment. This could be useful for other platforms too, e.g. you develop on macOS, cross-compile to Linux and would like to run your tests in Docker. Maybe the main entry point (what's currently called LinuxMain.swift) could set up the test environment, but then the entry point has to be compiled for the host architecture, while the test bundle would still need to be cross-compiled for the target architecture. SwiftPM might not support these "remote test runners", or however you call it, directly, but at least there should be some way to make it easier to test when cross-compiling.
3 Likes

LinuxMain.swift should be removed soon (or at least not required). Hopefully --enable-test-discovery becomes the default and LinuxMain.swift is no longer needed

5 Likes

That's totally cool, but until we have a timeline for when it will be removed, continued use of LinuxMain.swift on every platform other than Darwin seems weird. And renaming LinuxMain.swift to something more appropriate seems like a much easier task than making sure --enable-test-discovery works properly, at least in my understanding. Additionally, it could still make sense to have an explicit entry-point to accommodate for other test runners, especially when cross-compiling, as I described in my second point.

2 Likes

We're going to use @main attribute instead of main.swift file for start entry point of executable file.
Shall we also drop linuxmain/testmain.swift file support to comply with @main tag style using @testmain to indicate test entry point?
Just suggestions, hope to have consistent entry point naming convention across dev/test environment.

2 Likes

I was thinking along the same lines, but given that a test product shouldn't have access to the normal products' @main, wouldn't it be better to have a separate entry point for tests, and use @main in there as well?

2 Likes

LinuxMain.swift is basically dead anyway. I have used nothing but --enable-test-discovery for months now and it works just fine—even for Android. I use an external tool that flags if test coverage is incomplete, so I am reasonably certain no tests are being missed.


:+1: Some sort of plug‐in system for remote runners would be nice thing to have for Android too.

3 Likes

Indirect wish is robuster tooling support. Xcode crashes way too often on me due to some SPM related issue. The problem is that I cannot report those as these crashes are tied to org projects. I also cannot reduce the issue to some reproducible project which I could submit. The workaround I found is to open the manifest file add a random character to prevent Xcode from compiling next time, then open the main project, clear all SPM caches and remove the character.

One other wish: Updated Changelog file on the SPM repo. :sweat_smile:

6 Likes

The one valuable part of Apple’s opaque bug process is the ability to attach a project which will be seen by virtually no one. So you can always report direct rather than through swift.org.

The bug that crashes Xcode explicitly says to report it on bugs.swift.org, not through the feedback assistant. And as I mentioned above, it's not an option to just send Apple our whole project.

1 Like

My wish: I wish that SPM (and/or llbuild) would sandbox the compiler execution to ensure correct builds (a la Bazel). Concretely, I just had a really nasty issue where I forgot to specify a dependency in a reasonably complicated project which resulted in unreproducible compile errors.

This dangerous behavior can be easily reproduced in SwiftPM today, and have filed an issue: SR-12688.

Really, I hope this change happens ASAP. It’s incredibly annoying, and the difference in behaviour has allowed bugs to slip through on FOSS projects I have contributed to (I added tests on Darwin, they passed, but I forgot to update LinuxMain. Another contributor broke the test, but it didn’t run on their machine or the Linux-based CI we were running).

It’s possible to add infrastructure to your CI to mitigate against that, but you first have to hit and debug the problem before you remember that LinuxMain is even a thing.

It’s also super annoying when developing complex projects using test-driven development on Linux. Ideally you’d like to add tests as you go to validate what you’re writing, but you need to regenerate the file or remember to add the flag every time (VSCode’s Terminal window doesn’t have its own command history, so you may need to retype the test command when relaunching the editor).

What is stopping us removing LinuxMain and switching to discovery today?

1 Like