Thanks for the responses!
I think what you’re ultimately proposing boils down to having one or more “development configurations” which passes specific flags to SwiftPM’s various build stages.
Yes, that’s exactly what I proposed :) I used the word “local settings” but “development settings” is a much better way to name it. And the most important aspect of that proposal is that these development configurations can affect the build of targets that are not declared in the root package.
it looks like we need a way to compile a target and its dependencies with fuzzer (and friends) sanitizer enabled
The choice of which targets to sanitize is up to the user. I think that sometimes it may be better to sanitize less code, to avoid overwhelming the fuzzer with irrelevant coverage information.
.fuzzTarget
I also considered using something like .fuzzTarget , but found it more difficult to design, maybe someone here can help :) It might be helpful if I go through my initial design process here so that you see where I am coming from.
First, I think it will be useful to consider two examples when evaluating possible solutions. Both examples would be mixed language packages, but the first one, let’s call it p1_libFuzzer, would use libFuzzer and the other one, p1_swiftfuzz, would use a native Swift fuzzer. So they both have a target FuzzTest containing a main.swift file. But in that file, p1_libFuzzer defines the C entry point function LLVMFuzzerTestOneInput and p1_swiftfuzz calls a function defined by a Swift fuzzer package. They both depend on a package called P2:
let p2 = Package(
name: "P2",
targets: [
.target(name: "T2", dependencies: []), // Swift
.target(name: "T3", dependencies: []), // C
.target(name: "T4", dependencies: []) // extremely large, complex target that we don't want to sanitize
]
)
Now, I’ll try to see how a .fuzzTarget target type would work in terms of manifest API, usage, and implementation. Afterwards, I’ll list what I believe are the downsides of that approach.
Package API
A fuzzTarget would need, at least, a _sanitizers and _sanitized_targets argument.
let p1_libFuzzer = Package(
name: "P1_libFuzzer",
dependencies: [P2],
targets: [
.target(name: "T1", dependencies: ["T2", "T3", "T4"]),
.fuzzTarget(
name: "FuzzTest",
dependencies: ["T1"],
_sanitizers: [.address, .fuzzer],
_sanitized_targets: ["FuzzTest", "T1", "T2", "T3"],
swiftSettings: [
._parse_as_library
],
linkerSettings: [
._sanitize([.address, .fuzzer])
/*
link libFuzzer, although I am not sure it is right
the place to put that setting. The point is that
we can probably use SE-0238 to define some of the
settings that we need.
*/
]
)
]
)
let p1_swiftfuzz = Package(
name: "p1_swiftfuzz",
dependencies: [
P2,
swift_fuzzer
],
targets: [
.target(name: "T1", dependencies: ["T2", "T3", "T4"]),
.fuzzTarget(
name: "FuzzTest",
dependencies: ["T1", "swift_fuzzer"],
_sanitizers: [.fuzzer],
_sanitized_targets: ["FuzzTest", "T1", "T2", "T3"]
)
]
)
Although here p1_libFuzzer is kind of ugly. We could special-case libFuzzer support with something like _use_libFuzzer: true , which could also set a default value for _sanitizers and _sanitized_targets .
let p1_libFuzzer = Package(
name: "P1_libFuzzer",
dependencies: [P2],
targets: [
.target(name: "T1", dependencies: ["T2", "T3", "T4"]),
.fuzzTarget(
name: "FuzzTest",
dependencies: ["T1"],
_use_libFuzzer: true,
_sanitized_targets: ["FuzzTest", "T1", "T2", "T3"], // if _sanitized_targets was omitted, every target would be sanitized, including T4
)
]
)
Usage
Compiling FuzzTest, for either package, requires building a bunch of targets (T1, T2, T3) differently than a normal release build would. For that reason, I don’t think running swift build should automatically build FuzzTest, as it would compile the same targets multiple times and take a lot of time. That’s a bit confusing, but acceptable I think. So we would need a different command for building FuzzTest. Let’s say it is swift build --fuzzTarget FuzzTest. That command should then put its product in a different folder than .build/(debug|release)/, maybe in .build/FuzzTest-(debug|release)/ instead. So running a fuzz test would look like:
swift build -c release --fuzzTarget FuzzTest
.build/FuzzTest-release/FuzzTest
I don’t think it should be possible for a target in another package to depend on FuzzTest, because it would almost certainly result in conflicting build settings. Or, if it is possible, then its arguments (_sanitizers, _sanitized_targets, etc.) should be ignored. That’s also a bit confusing, but probably ok because I don’t see a reason to depend on FuzzTest anyway.
Implementation
In terms of implementation, I believe we need a couple new capabilities to support .fuzzTarget.
The first one is to add build settings to a target after all the dependencies have been resolved, and to do so only if these additional settings were specified by the root package.
And the second one is to define a build plan for a configuration that is neither debug nor release, but that is instead specific to a target.
Issues
First, it bothers me that a .fuzzTarget looks similar to a regular .target but behaves very differently.
- it is not built by
swift build automatically
- it doesn’t use the same cache as debug/release builds
- some of its settings are not applied if it is compiled as a dependency of another package
Moreover, we need to make a few decisions regarding what settings to allow, which may lead to an unusable API if we don’t think of everything. For example, should it be possible to use different sanitizers for different target dependencies? (I don’t know)
It can also be inconvenient for packages that contain many different fuzz tests, as the fuzz test targets will all use different build caches.
Then, if SwiftPM ever supports custom build configurations and a way to specify “development settings” as I proposed initially, this .fuzzTarget API will become redundant.
And finally, in terms of implementation, it seems like it is not more difficult to fully support the two features I initially proposed than it is to support .fuzzTarget. In a way, properly supporting .fuzzTarget boils down to creating new build configurations and adding development-only settings to a list of targets, which is what my initial pitch was about.
Development settings and custom build configurations
If I try to rewrite the packages p1_libFuzzer and p1_swiftfuzz using my initial solution, it should be easier to discuss the pros and cons of both approaches.
let p1_libFuzzer = Package(
name: "P1_libFuzzer",
dependencies: [P2],
targets: [
.target(name: "T1", dependencies: ["T2", "T3", "T4"]),
.target(
name: "FuzzTest",
dependencies: ["T1"],
swiftSettings: [._parse_as_library],
linkerSettings: [._sanitize([.address, .fuzzer])] // link libFuzzer
)
],
// maybe that argument should be split into _c_dev_setting, _swift_dev_setting, etc.
// it supports all the things that SE-0238 does
_development_settings: [
// maybe sanitize should be a language-agnostic setting
._swift(._sanitize([.address, .fuzzer]), forTargets: ["FuzzTest", "T1", "T2"], when: .config("fuzztest-release")),
._c(._sanitize([.address, .fuzzer]), forTargets: ["T3"], when: .config("fuzztest-release")),
]
)
let p1_swiftfuzz = Package(
name: "P1_swiftfuzz",
dependencies: [P2, swift_fuzzer],
targets: [
.target(name: "T1", dependencies: ["T2", "T3", "T4"]),
.target(name: "FuzzTest", dependencies: ["T1", "swift_fuzzer"])
],
_development_settings: [
._swift(._sanitize(.fuzzer), forTargets: ["FuzzTest", "T1", "T2"], when: .config("fuzztest-release")),
._c(._sanitize(.fuzzer), forTargets: ["T3"], when: .config("fuzztest-release")),
]
)
The obvious downside is that it is very verbose, especially for p1_libFuzzer. It is also seems difficult to use the ._sanitize option correctly, at least to me.
On the other hand, I think it is easier to understand how it works than .fuzzTarget. It is more flexible, and it should be more future-proof. For example, with that approach, I think it would also be possible to write and use a package providing an alternative to the Address Sanitizer (ASAN) in Swift.
Alternative
The options that are actually specified in this _development_settings argument are all ._sanitizers. Therefore, we could limit the scope of the proposal by replacing that argument with something like _sanitizer_settings without losing much.
let p1_libFuzzer = Package(
/* ... */,
_sanitizer_settings: [
([.address, .fuzzer], forTargets: ["FuzzTest", "T1", "T2"], when: .config("fuzztest-release"))
]
)
let p1_swiftfuzz = Package(
/* ... */,
_sanitizer_settings: [
([.fuzzer], forTargets: ["FuzzTest", "T1", "T2", "T3"], when: .config("fuzztest-release"))
]
)
Conclusion
I wanted to explain why I did not find an easy way to add fuzzer support and instead ended up pitching two features orthogonal to the problem of fuzz testing. Maybe I missed something, or maybe the problems I see with .fuzzTarget are not that bad.
Of course, the API would be somewhat simpler if we limited ourselves to libFuzzer support, but I think that would be a big mistake. To me, the most important feature is the ability to sanitize any target, even one that is declared in another package.
