Support for fuzz testers in Swift packages

Hi everyone!

Recently, I wrote a coverage-guided Swift fuzzer similar to libFuzzer, but I found that SwiftPM did not allow me to specify the build settings I needed, and I had to fork SwiftPM to make my project work.

Using a coverage-guided fuzzer requires the tested targets to be instrumented with LLVM’s SanitizerCoverage, which is kind of possible to do now with the option -sanitize=fuzzer when using a development version of swiftc. And it is possible to add that option to a package because SwiftPM supports custom build settings thanks to SE-0238, but only for targets that are declared in the root package, not for targets declared in a dependency.

Moreover, instrumenting a target makes its resulting binary unfit to be used for any other purpose than fuzzing, so I believe it would be best to isolate the products of a “fuzzing” build by using a separate build configuration than debug or release.

To make it more concrete, let’s say I have two packages P1 and P2:

let p1 = Package(
    name: "P1",
    dependencies: [
        .package(/* P2 */),
        .package(/* Fuzzer, a coverage-guided fuzzing engine */)
    ],
    targets: [
        .target(name: "T1", dependencies: ["T2"])
        .target(name: "FuzzTest", dependencies: ["T1", "Fuzzer"]),
    ]
)

let p2 = Package(
    name: "P2",
    dependencies: [],
    targets: [.target(name: "T2", dependencies: [])]
)

where the target FuzzTest is an executable and requires the targets FuzzTest, T1, and T2 (but not Fuzzer) to be compiled with -sanitize=fuzzer. There is currently no way to build it correctly with SwiftPM, because P1 cannot influence the way T2 is built. Even if there was a way to instrument T2 correctly, the build products would be placed in .build/release/ or .build/debug/, which is inconvenient.

My proposed solution has two parts:

  1. Allow the creation of new build configurations, other than debug or release. A new build configuration would be defined by its name only, and can be overlaid on top of an existing configuration. For example, thing-debug would inherit the settings of the debug configuration.

  2. Make it possible to modify the build settings of any target, even one that is not declared within the package. However, those custom build settings would only be “local” to the package. That is, they would not be applied if the package is built as a dependency. This is necessary to avoid conflicting build settings.

Together, these two features would allow one to rewrite the previous package P1 as:

let p1 = Package(
    name: "P1",
    dependencies: [/*unchanged*/],
    targets: [/*unchanged*/],
    _local_settings: [
        ._sanitize(._coverage, _for_targets: ["FuzzTest", "T1", "T2"], when: .configuration("fuzztest-release"))
    ]
)

and build it with:

swift build -c fuzztest-release

then launch the fuzz test executable with:

.build/fuzztest-release/FuzzTest

Xcode project generation would work as expected, with the build configuration fuzztest-release available and correctly configured.

I think that the features of “custom build configurations” and “local build settings” could be used for more than fuzz testing, although I have to admit I don’t have any other example right now.

I have started working on an implementation here. It supports custom build settings, a way to add the -sanitize=fuzzer flag in the _local_settings, and Xcode project generation. While it is very incomplete and buggy, so far it seems like there aren’t blocking issues. I was even able to use it to test my fuzzer.

There are many details to discuss, and many different ways to support the use of fuzz testers. Maybe a solution that is more specific to my use case would be better, or maybe these kinds of settings should not even be part of the package manifest. But I hope we can agree that supporting sanitizer tools written as Swift packages is a worthy goal. In particular, I am very excited about the prospect of native Swift fuzzers.

Please let me know if you agree with my proposed solution, or if you believe that a completely different approach would be better, or if the problem is not worth solving. If we agree on a certain approach, I can prepare a more detailed pitch :blush:

4 Likes

CC @lukasa, @Aciid & @NeoNacho

I am profoundly in favour of adding support for building fuzzers in SwiftPM. I've been playing with Swift's libFuzzer support on and off for more than a year now, and would love more formal support.

My reading of your proposal is that you need things that are related, but somewhat different, than what libFuzzer requires. libFuzzer requires that we compile with -sanitize=address,fuzzer and that we provide the specific magic libFuzzer entry point, as well as compiling with -parse-as-library (to prevent swiftc moaning about the lack of a func main and to prevent SwiftPM moaning about the lack of a main.swift).

I think what you're ultimately proposing boils down to having one or more "development configurations" which passes specific flags to SwiftPM's various build stages. I think this is a good idea, worth exploring, but the SwiftPM folks would definitely need to weigh in. I'd also like to stress that whatever we land on I think it is critical that SwiftPM be able to invoke the sanitisers on all supported languages: that is, we should be passing -fsanitize=address,fuzzer to C code as well as just to Swift code. After all, the C code in mixed-source projects is where the real danger lies!

Thanks for the writing up a pitch! Adding fuzzing support to SwiftPM seems like a great enhancement. I think we should take a step back and try to look at the requirements for fuzz testing. Features like build configuration and advanced build settings are definitely future goals of SwiftPM but they seem orthogonal to fuzzing support. I briefly read the libFuzzer documentation and it looks like we need a way to compile a target and its dependencies with fuzzer (and friends) sanitizer enabled. We also need a fuzz target that will act as the entry point. This is very similar to how SwiftPM's test targets work. One approach could be adding a .fuzzTarget API SwiftPM to define these types of targets. This will also allow us to resolve the problems -parse-as-library and main.swift issues that @lukasa mentioned. An alternative approach could be requiring to pass the fuzz target's name on the command using SwiftPM's --target option whenever fuzzing is enabled. It would be great if you're interested in fleshing out a concrete proposal!

PS: I am moving this post to SwiftPM's development category as that is a more appropriate place for this discussion.

Thanks for the responses!


I think what you’re ultimately proposing boils down to having one or more “development configurations” which passes specific flags to SwiftPM’s various build stages.

Yes, that’s exactly what I proposed :) I used the word “local settings” but “development settings” is a much better way to name it. And the most important aspect of that proposal is that these development configurations can affect the build of targets that are not declared in the root package.

it looks like we need a way to compile a target and its dependencies with fuzzer (and friends) sanitizer enabled

The choice of which targets to sanitize is up to the user. I think that sometimes it may be better to sanitize less code, to avoid overwhelming the fuzzer with irrelevant coverage information.


.fuzzTarget

I also considered using something like .fuzzTarget , but found it more difficult to design, maybe someone here can help :) It might be helpful if I go through my initial design process here so that you see where I am coming from.

First, I think it will be useful to consider two examples when evaluating possible solutions. Both examples would be mixed language packages, but the first one, let’s call it p1_libFuzzer, would use libFuzzer and the other one, p1_swiftfuzz, would use a native Swift fuzzer. So they both have a target FuzzTest containing a main.swift file. But in that file, p1_libFuzzer defines the C entry point function LLVMFuzzerTestOneInput and p1_swiftfuzz calls a function defined by a Swift fuzzer package. They both depend on a package called P2:

let p2 = Package(
	name: "P2",
	targets: [
		.target(name: "T2", dependencies: []), // Swift
		.target(name: "T3", dependencies: []), // C
		.target(name: "T4", dependencies: []) // extremely large, complex target that we don't want to sanitize
	]
)

Now, I’ll try to see how a .fuzzTarget target type would work in terms of manifest API, usage, and implementation. Afterwards, I’ll list what I believe are the downsides of that approach.

Package API

A fuzzTarget would need, at least, a _sanitizers and _sanitized_targets argument.

let p1_libFuzzer = Package(
	name: "P1_libFuzzer",
	dependencies: [P2],
	targets: [
		.target(name: "T1", dependencies: ["T2", "T3", "T4"]),
		.fuzzTarget(
			name: "FuzzTest", 
			dependencies: ["T1"], 
			_sanitizers: [.address, .fuzzer],
			_sanitized_targets: ["FuzzTest", "T1", "T2", "T3"],
			swiftSettings: [
				._parse_as_library
			],
			linkerSettings: [
				._sanitize([.address, .fuzzer]) 
				/*
				link libFuzzer, although I am not sure it is right
				the place to put that setting. The point is that 
				we can probably use SE-0238 to define some of the 
				settings that we need.
				*/
			]
		)
	]
)
let p1_swiftfuzz = Package(
	name: "p1_swiftfuzz",
	dependencies: [
		P2,
		swift_fuzzer
	],
	targets: [
		.target(name: "T1", dependencies: ["T2", "T3", "T4"]),
		.fuzzTarget(
			name: "FuzzTest", 
			dependencies: ["T1", "swift_fuzzer"], 
			_sanitizers: [.fuzzer],
			_sanitized_targets: ["FuzzTest", "T1", "T2", "T3"]
		)
	]
)

Although here p1_libFuzzer is kind of ugly. We could special-case libFuzzer support with something like _use_libFuzzer: true , which could also set a default value for _sanitizers and _sanitized_targets .

let p1_libFuzzer = Package(
	name: "P1_libFuzzer",
	dependencies: [P2],
	targets: [
		.target(name: "T1", dependencies: ["T2", "T3", "T4"]),
		.fuzzTarget(
			name: "FuzzTest", 
			dependencies: ["T1"], 
			_use_libFuzzer: true,
			_sanitized_targets: ["FuzzTest", "T1", "T2", "T3"], // if _sanitized_targets was omitted, every target would be sanitized, including T4
		)
	]
)

Usage

Compiling FuzzTest, for either package, requires building a bunch of targets (T1, T2, T3) differently than a normal release build would. For that reason, I don’t think running swift build should automatically build FuzzTest, as it would compile the same targets multiple times and take a lot of time. That’s a bit confusing, but acceptable I think. So we would need a different command for building FuzzTest. Let’s say it is swift build --fuzzTarget FuzzTest. That command should then put its product in a different folder than .build/(debug|release)/, maybe in .build/FuzzTest-(debug|release)/ instead. So running a fuzz test would look like:

swift build -c release --fuzzTarget FuzzTest
.build/FuzzTest-release/FuzzTest

I don’t think it should be possible for a target in another package to depend on FuzzTest, because it would almost certainly result in conflicting build settings. Or, if it is possible, then its arguments (_sanitizers, _sanitized_targets, etc.) should be ignored. That’s also a bit confusing, but probably ok because I don’t see a reason to depend on FuzzTest anyway.

Implementation

In terms of implementation, I believe we need a couple new capabilities to support .fuzzTarget.
The first one is to add build settings to a target after all the dependencies have been resolved, and to do so only if these additional settings were specified by the root package.
And the second one is to define a build plan for a configuration that is neither debug nor release, but that is instead specific to a target.

Issues

First, it bothers me that a .fuzzTarget looks similar to a regular .target but behaves very differently.

  • it is not built by swift build automatically
  • it doesn’t use the same cache as debug/release builds
  • some of its settings are not applied if it is compiled as a dependency of another package

Moreover, we need to make a few decisions regarding what settings to allow, which may lead to an unusable API if we don’t think of everything. For example, should it be possible to use different sanitizers for different target dependencies? (I don’t know)

It can also be inconvenient for packages that contain many different fuzz tests, as the fuzz test targets will all use different build caches.

Then, if SwiftPM ever supports custom build configurations and a way to specify “development settings” as I proposed initially, this .fuzzTarget API will become redundant.

And finally, in terms of implementation, it seems like it is not more difficult to fully support the two features I initially proposed than it is to support .fuzzTarget. In a way, properly supporting .fuzzTarget boils down to creating new build configurations and adding development-only settings to a list of targets, which is what my initial pitch was about.

Development settings and custom build configurations

If I try to rewrite the packages p1_libFuzzer and p1_swiftfuzz using my initial solution, it should be easier to discuss the pros and cons of both approaches.

let p1_libFuzzer = Package(
	name: "P1_libFuzzer",
	dependencies: [P2],
	targets: [
		.target(name: "T1", dependencies: ["T2", "T3", "T4"]),
		.target(
			name: "FuzzTest", 
			dependencies: ["T1"],
			swiftSettings: [._parse_as_library],
			linkerSettings: [._sanitize([.address, .fuzzer])] // link libFuzzer
		)
	],
	// maybe that argument should be split into _c_dev_setting, _swift_dev_setting, etc.
	// it supports all the things that SE-0238 does
	_development_settings: [
		// maybe sanitize should be a language-agnostic setting
		._swift(._sanitize([.address, .fuzzer]), forTargets: ["FuzzTest", "T1", "T2"], when: .config("fuzztest-release")),
		._c(._sanitize([.address, .fuzzer]), forTargets: ["T3"], when: .config("fuzztest-release")),
	]
)

let p1_swiftfuzz = Package(
	name: "P1_swiftfuzz",
	dependencies: [P2, swift_fuzzer],
	targets: [
		.target(name: "T1", dependencies: ["T2", "T3", "T4"]),
		.target(name: "FuzzTest", dependencies: ["T1", "swift_fuzzer"])
	],
	_development_settings: [ 
		._swift(._sanitize(.fuzzer), forTargets: ["FuzzTest", "T1", "T2"], when: .config("fuzztest-release")),
		._c(._sanitize(.fuzzer), forTargets: ["T3"], when: .config("fuzztest-release")),
	]
)

The obvious downside is that it is very verbose, especially for p1_libFuzzer. It is also seems difficult to use the ._sanitize option correctly, at least to me.

On the other hand, I think it is easier to understand how it works than .fuzzTarget. It is more flexible, and it should be more future-proof. For example, with that approach, I think it would also be possible to write and use a package providing an alternative to the Address Sanitizer (ASAN) in Swift.

Alternative

The options that are actually specified in this _development_settings argument are all ._sanitizers. Therefore, we could limit the scope of the proposal by replacing that argument with something like _sanitizer_settings without losing much.

let p1_libFuzzer = Package(
	/* ... */,
	_sanitizer_settings: [
		([.address, .fuzzer], forTargets: ["FuzzTest", "T1", "T2"], when: .config("fuzztest-release"))
	]
)

let p1_swiftfuzz = Package(
	/* ... */,
	_sanitizer_settings: [
		([.fuzzer], forTargets: ["FuzzTest", "T1", "T2", "T3"], when: .config("fuzztest-release"))
	]
)

Conclusion

I wanted to explain why I did not find an easy way to add fuzzer support and instead ended up pitching two features orthogonal to the problem of fuzz testing. Maybe I missed something, or maybe the problems I see with .fuzzTarget are not that bad.

Of course, the API would be somewhat simpler if we limited ourselves to libFuzzer support, but I think that would be a big mistake. To me, the most important feature is the ability to sanitize any target, even one that is declared in another package.

I don't think that I agree with this position. My experience of fuzzing is that fuzzing improves monotonically with additional instrumentation. Code that is linked into your application affects the correctness properties of your application, and the fuzzer being unable to see it simply means that it is not able to fuzz as effectively. I've got similar feelings regarding all sanitisers: the more code you can sanitise, the better the effectiveness of the sanitiser.

With that in mind, I propose removing from scope the idea of only sanitising specific targets, and instead saying that sanitiser settings should affect all dependencies built by swift package manager. Obviously if you are linking against prebuilt binary libraries then there is nothing that the Swift package manager can do, but otherwise I think we should be as broad-scoped as possible.

Removing this from scope greatly simplifies matters, as we are now talking about applying package-wide build settings. We can then say that the setting .fuzzTarget implies passing --sanitize=fuzzer to the build when building that target. We can also say that it excludes the fuzz target from regular build invocations, and that it makes the fuzz target ineligible for being used from products.

Heck, we can even add a .fuzzer product that requires that its top-level targets be fuzzTargets.

There is no need for new flags for building fuzz targets: swift build already has a --target flag that can be used here.

If we forbid other packages from depending on fuzzTargets this is not a concern.

Indeed. I think the only meaningful settings for a fuzzTarget are:

  1. A list of sanitisers in addition to libFuzzer instrumentation that they require, if any.
  2. Whether they intend to use libFuzzer itself for fuzzing (that is, do they need parse-as-library and the appropriate linkage).

Removing this from scope greatly simplifies matters, as we are now talking about applying package-wide build settings. We can then say that the setting .fuzzTarget implies passing --sanitize=fuzzer to the build when building that target. We can also say that it excludes the fuzz target from regular build invocations, and that it makes the fuzz target ineligible for being used from products.

Heck, we can even add a .fuzzer product that requires that its top-level targets be fuzzTarget s.

I am not entirely convinced that it is always worth sanitizing more code. But I can admit that in most cases, selective target sanitization is not needed, and that it is not worth supporting in SwiftPM. Indeed, removing that assumption greatly simplifies everything. For some reason I had never questioned it. I'll make a simplified pitch later :slight_smile:

I’ve been thinking more about the benefits of supporting fuzzing in SwiftPM. In my initial pitch, I had two advantages in mind (caching via new build configurations + selective sanitization), but now I agree that these things are not that important. Then, the possibility of a .fuzzTarget was mentioned, and I became convinced it was a better idea. But now I don’t actually see the benefits that a fuzzTarget would bring.
What does a special target or product type provide that calling swift build --sanitize fuzzer doesn't? And are these benefits worth the extra complexity?

In my opinion, what is really needed to use libfuzzer are these two minor changes, which I hope would not require any formal proposal:

  • shipping libFuzzer with Swift. For now, libFuzzer can only be used on development versions of the compiler. (not true, it seems like it was a bug in my setup)
  • Allowing --sanitize fuzzer to be passed to swift build, I can submit a PR for that if needed

Then using libFuzzer would look like:

let package = Package(
    name: "P",
    dependencies: [/*...*/],
    targets: [
        .target(
            name: "FuzzTest", 
            dependencies: [/*...*/], 
            swiftSettings: [.unsafeFlags(["--parse-as-library"])]
        )
    ]
)
/* usage
swift build --sanitize fuzzer --sanitize address
.build/debug/FuzzTest
*/

And for supporting third-party fuzzers, these three small changes are needed:

  • allowing the -sanitize-coverage option to be used without a -sanitizer. As far as I can tell, the only reason it is not currently allowed is for user-friendliness.
  • add a --sanitize-coverage option to SwiftPM. I can also submit a PR for that.
  • provide a way to create a target in SwiftPM that is never sanitized (because sanitizing a sanitizer causes an infinite loop). Ideally, none of its dependencies would be sanitized either, but that's more complicated and not strictly necessary. It could be done with an additional parameter to .target or with a new .fuzzer target.

Using a third-party fuzzer would look like:

let fuzzerPackage = Package(
    name: "Fuzzer",
    dependencies: [/*...*/],
    targets: [
        .target(
            name: "Fuzzer", 
            dependencies: [/*...*/], 
            _never_sanitize: true
        )
    ]
)

let p = Package(
    name: "FuzzTest",
    dependencies: [Fuzzer, /*...*/],
    targets: [
        .target(
            name: "FuzzTest", 
            dependencies: ["Fuzzer", /*...*/]
        )
    ]
)


/* usage
swift build --sanitize-coverage edge,pc-guard,trace-cmp,indirect-calls
.build/debug/FuzzTest
*/

I think these changes are much smaller and less controversial. The only problematic one is the ability to create a target (or product) that is never sanitized, which I would really like to have but is not needed for libFuzzer support.

Would these changes be sufficient to you? If not, what is missing?

The biggest issue is that by default when you run swift build in the directory of a project, SwiftPM builds all targets. That means that you will be building your libFuzzer targets always. That doesn't work well unless libfuzzer is also linked in, as the produced binary ends up not having an entry point.

That's really my only concern though: otherwise I'd personally be perfectly happy with your proposal.

Is there any chance that the fuzzer could be integrated with the future Package Manager Extensible Build Tools feature?

A .fuzzTarget is needed if we want to add real support for libFuzzer in SwiftPM otherwise SwiftPM doesn't know when and how to build the entry point fuzz target. Unsafe flag is an escape hatch to let you do experimentation and not something that users should rely on for supported features. I guess the fuzzing should already work by passing fuzzer flag using -X* flags and then using the unsafe flag escape hatch.

I don't think we can add ability to selectively exclude targets from fuzzing without the full build settings model (requires heavy design work).

A .fuzzTarget is needed if we want to add real support for libFuzzer in SwiftPM otherwise SwiftPM doesn't know when and how to build the entry point fuzz target. Unsafe flag is an escape hatch to let you do experimentation and not something that users should rely on for supported features. I guess the fuzzing should already work by passing fuzzer flag using -X* flags and then using the unsafe flag escape hatch.

Maybe it's best not to add real support for libFuzzer. I am really against a .fuzzTarget API as I think its behavior would be strange and complex and it would allow only a very narrow range of use cases. Maybe it can be done well, but I would prefer if someone else figured it out.

In the meantime, the one change that I need for my use case that is absolutely necessary is to allow using the -sanitize-coverage option of swiftc without any sanitizer enabled, so I will file a bug for that. It wouldn't break any existing code and will allow writing sanitizer hooks in Swift. I can work around every other limitation of Swift/SwiftPM regarding fuzzer support.