Passing a MutableSpan to an annotated C function

Using Swift
, ,
1

Hello!

I'm trying to interoperate with C in a safe way by annotating the API to generate safe wrappers but I don't know how to pass a mutable span.

To make my issue obvious I created this super simple C module:

int mini_sum(const unsigned char *in32);
int mini_fill(unsigned char *out64, const unsigned char *in32);

With these annotations:

---
Name: CMini
Functions:
- Name: mini_sum
  Parameters:
  - Position: 0
    NoEscape: true
    BoundsSafety: { Kind: counted_by, BoundedBy: 32 }
- Name: mini_fill
  Parameters:
  - Position: 0
    NoEscape: true
    BoundsSafety: { Kind: counted_by, BoundedBy: 64 }
  - Position: 1
    NoEscape: true
    BoundsSafety: { Kind: counted_by, BoundedBy: 32 }

Taking a look at what that generates…

xcrun swift-synthesize-interface \
  -I Sources/CMini/include \                 
  -Xcc -fmodule-map-file=Sources/CMini/include/module.modulemap \    
  -module-name CMini \    
  -target arm64-apple-macos27 \
  -sdk $(xcrun --sdk macosx --show-sdk-path)

I get these result (relevant part):

public func mini_sum(_ in32: Span<UInt8>) -> Int32

@_lifetime(out64: copy out64)
public func mini_fill(_ out64: inout MutableSpan<UInt8>, _ in32: Span<UInt8>) -> Int32

So far great. I can call mini_sum with a span and not get any safety warnings!

import CMini
let input = [UInt8](repeating: 7, count: 32)

let n = mini_sum(input.span)

But calling mini_fill in similar fashion has proved impossible with a wack-a-mole of errors:

var out = [UInt8](repeating: 0, count: 64)
var mutableSpan = out.mutableSpan // Lifetime-dependent value escapes its scope
_ = mini_fill(&mutableSpan, input) // Expression uses unsafe constructs but is not marked with 'unsafe'
_ = consume mutableSpan // Missing reinitialization of inout parameter 'mutableSpan' after consume while accessing memory

The example above is just one of the many iterations I have tried which include everything from OutputSpan, to MutableSpan(mutating:) initializer.. nothing seems to fit.

I can compile if I just pass a MutableSpan<UInt8>() but that clearly crashes on runtime. I also suspect the compiler may not be picking up the right overload for the function... although it shows up in the generated wrapper so I don't know.

Can anyone point me in the right direction? Or at least show me how to create a MutableSpan in Swift 6.4.

Thanks!!

1 Like
2

Quick update, I managed to call the correct C overload unsafely with:

import CMini
let input = [UInt8](repeating: 7, count: 32)
var output = [UInt8]()

withUnsafeTemporaryAllocation(of: UInt8.self, capacity: 64) { buffer in
    var mutableSpan = unsafe MutableSpan(_unsafeElements: buffer)
    _ = mini_fill(&mutableSpan, input.span)
    for e in mutableSpan {
        output.append(e)
    }
}

But when I try to do that safely with:

withTemporaryAllocation(of: UInt8.self, capacity: 64) { outputSpan in
    var mutableSpan = outputSpan.mutableSpan
    _ = mini_fill(&mutableSpan, input.span)
    for e in mutableSpan {
        output.append(e)
    }
}

It compiles without errors but crashes with an assertion failure for checked bounds:

/Users/d/Developer/CraigWrong/span-repro/Sources/CMini/include/cmini.h:5: Fatal error: bounds check failure in mini_fill: expected 64 but got 0

💣 Program crashed: System trap at 0x00000001a6c5bd4c

Platform: arm64 macOS 27.0 (26A5425a)

Thread 0 crashed:

0 0x00000001a6c5bd4c _assertionFailure(_:_:file:line:flags:) + 216 in libswiftCore.dylib
1 mini_fill(_:_:) + 596 in App at /Users/d/Developer/CraigWrong/@__swiftmacro_So9mini_fill15_SwiftifyImportfMp_.swift:4:7
2 closure #1 in  + 376 in App at /Users/d/Developer/CraigWrong/span-repro/Sources/App/main.swift:21:9

    19│ withTemporaryAllocation(of: UInt8.self, capacity: 64) { outputSpan in
    20│     var mutableSpan = outputSpan.mutableSpan
    21│     _ = mini_fill(&mutableSpan, input.span)                                                                                                                                                          
      │         ▲
    22│     for e in mutableSpan {
    23│         output.append(e)

3 main + 240 in App at /Users/d/Developer/CraigWrong/span-repro/Sources/App/main.swift:19:1

    17│ //}
    18│ 
    19│ withTemporaryAllocation(of: UInt8.self, capacity: 64) { outputSpan in                                                                                                                                
      │ ▲
    20│     var mutableSpan = outputSpan.mutableSpan
    21│     _ = mini_fill(&mutableSpan, input.span)

Backtrace took 0.08s
3

Answering your follow-up question: OutputSpan.mutableSpan returns a span over the initialized elements, which means it will have length 0 in your case (because the output span starts fully uninitialized).

Least unsafe way to address this: I'm not sure. I don't know myself if apinotes support an "output only mode", and if not you'll have to do some kind of "assume these elements are already initialized" or manually set them to 0 or something before getting the span. But then you're getting back to your first example with the array, and I'm not actually sure why that one doesn't work.

1 Like
4

That was it! Thanks!!

I don't know why the first example does not work but now we have a 100% safe way to call C functions with in-out arrays which is amazing:

var output = [UInt8](capacity: 64) { outputSpan in
    outputSpan.append(repeating: 0, count: outputSpan.freeCapacity)
    var mutableSpan = outputSpan.mutableSpan
    _ = mini_fill(&mutableSpan, input.span)
}

EDIT: I replaced withTemporaryAllocation with Array(capacity:)which makes it more concise. Still need to fill up the span before we pass it along.

5

More insight into the errors I was getting.

Seems like the function receiving an extra unsafe pointer somehow affects the lifespan of the mutable span making it not compile.

This function from libsecp256k1 illustrates the issue:

SECP256K1_API int secp256k1_ecdsa_signature_serialize_compact(
    const secp256k1_context *ctx,
    unsigned char *output64,
    const secp256k1_ecdsa_signature *sig
) SECP256K1_ARG_NONNULL(1) SECP256K1_ARG_NONNULL(2) SECP256K1_ARG_NONNULL(3);

With the following annotation:

- Name: secp256k1_ecdsa_signature_serialize_compact
  SwiftName: "secp256k1_context.ecdsaSignatureSerializeCompact(self:into:_:)"
  Parameters:
  - Position: 0          # ctx
    Nullability: N
  - Position: 1          # output64
    NoEscape: true
    BoundsSafety: { Kind: counted_by, BoundedBy: 64 }
    Nullability: N
  - Position: 2          # sig
    NoEscape: true
    Nullability: N

Notice that the first parameter is synthesized as self and for some reason that doesn't affect safety. But the last parameter is an UnsafePointer.

Here's what that generates (as a method of context):

@_lifetime(output64: copy output64)
public final func ecdsaSignatureSerializeCompact(into output64: inout MutableSpan<UInt8>, _ sig: UnsafePointer<secp256k1_ecdsa_signature>) -> Int32

This somehow impeded passing a MutableSpan for the middle parameter.

var sig: secp256k1_ecdsa_signature = …
let out = [UInt8](capacity: 64) { outputSpan in
    outputSpan.append(repeating: 0, count: outputSpan.freeCapacity)
    var mutableSpan = outputSpan.mutableSpan // ERROR: Lifetime-dependent variable 'mutableSpan' escapes its scope
    _ = raw.ecdsaSignatureSerializeCompact(into: &mutableSpan, &sig) // WARN: Expression uses unsafe constructs but is not marked with 'unsafe'
}

The solution would be to annotate that parameter with __single so Swift can generate a borrow or inout parameter. Unfortunately the synthesizer cannot handle that at the moment:

<API Notes>:1:44: error: unparsed tokens following type
const secp256k1_ecdsa_signature * _Nonnull _single

So the only workaround I found is to mark is as a Span of 1:

  - Position: 2          # sig
    NoEscape: true
    BoundsSafety: { Kind: counted_by, BoundedBy: 1 }
    Nullability: N

And call it with a UniqueBox:

let out = [UInt8](capacity: 64) { outputSpan in
    outputSpan.append(repeating: 0, count: outputSpan.freeCapacity)
    var mutableSpan = outputSpan.mutableSpan
    _ = raw.ecdsaSignatureSerializeCompact(into: &mutableSpan, UniqueBox(sig).span)
}

Which leaves me a bit concerned about efficiency but at least is memory safe guaranteed.

Anyway, any input on the matter I'd really appreciate.

Cheers!

6

Responding to myself here. So the ultimate way to call safe C function wrappers seems to require UniqueArray and UniqueBox.

This way all closures disappear.

Example:

let signature = …
var serializedSignature = UniqueArray(repeating: UInt8(0), count: 64)
var serializedSignatureSpan = serializedSignature.mutableSpan
serializeSignature(UniqueBox(signature).span, into: &serializedSignatureSpan)

Now if only there was away to efficiently convert the UniqueArray into a regular array when the elements are known to be copyable :thought_balloon:

7

Slightly off topic, and I don't know if the importer recognizes this, but C has supported annotating fixed size array parameters since c99. So you can declare these functions as:

int mini_sum(const unsigned char in[static 32]);
int mini_fill(unsigned char out[static 64], const unsigned char in[static 32]);

Recent versions of Clang and GCC even perform some static analysis based on this.

It doesn't change much on the Swift side, but I think it makes it much more obvious on the C side how big the array parameters are.

1 Like
8

I believe you could replace UniqueArray with Array there and it would work the same way. Does it not?

And the UniqueBox does seem to be currently necessary but there is a pitch for Span(ofOne: foo) which will do the same thing a little more directly.

1 Like
9

Well I'll be damned! The compiler was giving me some errors yesterday which is why I started this whole thread but you're right! It works with Array just the same! :man_facepalming:

As for Span(ofOne:) that's great. I guess I could make myself a replacement using UniqueBox internally until that lands.

The final solution would be for the synthesized methods to allow __single annotations which they can turn into borrow or inout directly eliminating the need for single element spans.

As for what @Nobody1707 mentioned, that's a case for generating Inline or RigidArray parameters but I'm not sure if we would need spans regardless.

10

I'm not sure what is the type of your signature, but if it's copyable then CollectionOfOne.span would work as well without provoking a heap allocation like UniqueBox.

2 Likes
11

Just tried it, worked like a charm. Thank you!!

1 Like