Memory usage and performance of a function in a package

I'm working on a Swift package that contains an exponential function as shown below:

import Accelerate

public func exp(_ vector: Vector<Float>) -> Vector<Float> {
    let result = vForce.exp(vector.values)
    let vec = Vector(result)
    return vec
}

I can also write the function as shown here:

public func exp(_ vector: Vector<Float>) -> Vector<Float> {
    var vec = Vector<Float>(length: vector.length, fill: 0)
    vForce.exp(vector.values, result: &vec.values)
    return vec
}

How can I measure the memory usage and performance for each of these functions? I'm aware of Instruments in Xcode but it seems to be restricted to profiling an entire app. Can Instruments be used to profile a Swift package or is there some other tool I should use?

One way or another you'll need an actual test driver, to specify the actual data and usage pattern of this function. You can't really [empirically] analyse its performance in abstract.

One good way to write such a driver is with the Benchmark package.

And then given that benchmark, you can run it under Instruments.

I wouldn't gauge CPU performance while running Instruments, as Instruments has some overhead which can perturb your benchmark. Instruments is for active analysis of code's performance / behaviour. For getting your actual performance scores, to answer your question about which implementation is superior, you just run it via the Benchmark package (with swift package benchmark) and it does a clean execution of the release build.

Tangentially, don't forget to mark your function as @inlinable, as otherwise the overhead of the function call itself will likely dominate your execution time.

(functions are implicitly inlinable when used within the same module, but if used from another module they are not unless explicitly annotated as such)

1 Like

I went through the XCTest documentation and it supports Performance Tests which provide various metrics such as CPU, memory, and elapsed time. This looks like a much easier solution than relying on an entire Benchmark package.

Unless you want to measure on Linux or need a way to check it systematically in CI or want additional metrics not provided by xctest, or…. (Admittedly biased as the main author of the benchmark package :slight_smile: )

2 Likes

The Benchmark package must be added as a dependency. But if I don't want users of my package to run the benchmarks then how do I exclude this dependency? I'm not aware of any Swift package features that support optional dependencies.

It would only be a dependency of your benchmark target, so they wouldn’t build it unless they ran benchmarks.

The best practice now is to just have a separate package inside the package that references to parent package by path.

Good examples you can look at are eg:

or

Check out their Benchmarks folder and their Package.swift there. Then no dependency on the Benchmark package is added to the top level package, so clients depending on it are completely unaffected. It also means the benchmarks can push requirements to newer platforms/toolchains.

2 Likes

This shouldn't matter at all here, where he's computing exp(x) for each element of a vector. Even for scalar exp() function call overhead is pretty minimal (it takes maybe 15-30 instructions for a good implementation to compute a single-precision exp, depending on the target), but when you're amortizing it over the elements of a vector it goes away entirely.

1 Like