Benchmarking Swift NIO and other Server Setups like Vapor

Related Projects Benchmark
1

Has anyone attempted to use Benchmark to profile Swift NIO, Vapor and related APIs yet? I'm attempting to bind a server, but when running via swift package benchmark, I'm getting the following error: failed with bind(descriptor:ptr:bytes:): Operation not permitted (errno: 1).

My only hunch is macOS sandboxing is getting in the way, blocking incoming connections, but I could very much be wrong here.

Note that I am able to run the executable directly, but the log output is not nearly as comprehensible, so I'm hoping there is an easy workaround here:

2024-07-19T03:38:42-0700 notice codes.vapor.application : [Vapor] Server started on http://127.0.0.1:56121
2024-07-19T03:38:42-0700 info codes.vapor.application : request-id=000DB1C5-2EB4-4702-AD52-483510088632 [Vapor] GET /resource
  0% [                                                            ] ETA: 00:00:00 | :SingleRequest_AsyncHTTPClient_TCP_Loopback_TCP_HTTP_Vapor_Memory_1GB_Random
2024-07-19T03:38:48-0700 notice codes.vapor.application : [Vapor] Server started on http://127.0.0.1:56152
100% [------------------------------------------------------------] ETA: 00:00:00 | :SingleRequest_AsyncHTTPClient_TCP_Loopback_TCP_HTTP_Vapor_Memory_1GB_Random
Debug results for SingleRequest_AsyncHTTPClient_TCP_Loopback_TCP_HTTP_Vapor_Memory_1GB_Random:

Time (wall clock):
       Value     Percentile TotalCount 1/(1-Percentile)

4680843263.000 0.000000000000          1           1.00
4680843263.000 1.000000000000          1
#[Mean    = 4678746112.000, StdDeviation   =        0.000]
#[Max     = 4680843263.000, Total count    =            1]
#[Buckets =           23, SubBuckets     =         2048]


Time (total CPU):
       Value     Percentile TotalCount 1/(1-Percentile)

5599395839.000 0.000000000000          1           1.00
5599395839.000 1.000000000000          1
#[Mean    = 5597298688.000, StdDeviation   =        0.000]
#[Max     = 5599395839.000, Total count    =            1]
#[Buckets =           23, SubBuckets     =         2048]


Memory (resident peak):
       Value     Percentile TotalCount 1/(1-Percentile)

3800039423.000 0.000000000000          1           1.00
3800039423.000 1.000000000000          1
#[Mean    = 3798990848.000, StdDeviation   =        0.000]
#[Max     = 3800039423.000, Total count    =            1]
#[Buckets =           22, SubBuckets     =         2048]


Malloc (total):
       Value     Percentile TotalCount 1/(1-Percentile)

 3934207.000 0.000000000000          1           1.00
 3934207.000 1.000000000000          1
#[Mean    =  3933184.000, StdDeviation   =        0.000]
#[Max     =  3934207.000, Total count    =            1]
#[Buckets =           20, SubBuckets     =         2048]


Instructions:
       Value     Percentile TotalCount 1/(1-Percentile)

28185722879.000 0.000000000000          1           1.00
28185722879.000 1.000000000000          1
#[Mean    = 28177334272.000, StdDeviation   =        0.000]
#[Max     = 28185722879.000, Total count    =            1]
#[Buckets =           25, SubBuckets     =         2048]

Sample code is available here: GitHub - mochidev/swift-server-benchmarks

2

errno value 1 is indeed EPERM (Operation not permitted) and that's what a sandbox violation would give you.

Have you tried swift package --disable-sandbox benchmark ... ?

1 Like
3

I didn't know that was an option! Seems like this now works, thanks!

1 Like
4

@dimi I'm very interested in this topic. What is your angle, personal interest?

5

I mostly wanted to know if I was shooting myself in the foot any more than I already was by enabling features like response compression, but that further lead to wanting to identify where in the pipeline things are slower so we have some targets we can improve as a community. I have the start working over here if you want to poke around and contribute: GitHub - mochidev/swift-server-benchmarks

1 Like
6

Yep, for future googlers this is covered here if you’ scroll down to network and disk permissions:

https://swiftpackageindex.com/ordo-one/package-benchmark/1.23.5/documentation/benchmark/runningbenchmarks

2 Likes