C-to-Swift function calls, are they glued? Specifically, CGEvent.copy()?

[JetForMe (Rick M)]

I reported this to Apple with a reproducing project as FB13297964, but I have questions.

TL;dr

How does CGEvent.copy() differ from CGEventCreateCopy()? Is it just syntactic sugar? Is there some glue code that retains or releases? Is it generated by the compiler, or provided by Apple (i.e. written by a human and possibly buggy)?

Detail

I recently ran into an issue when I converted some working CGEventTap code from C to Swift. A CGEventTap is created in C with CGEventTapCreate() and passed a pointer to a C callback function. The callback is passed the incoming event, and you can return that event, or a new one, to be handled by the rest of the system.

My C code worked fine. But when translated to Swift, I was getting “Invalid event returned by callback” log messages (and my events were ignored). After investigating, I discovered that CGEventCreateCopy() creates a CGEvent with a retain count of 3, while CGEvent.copy() creates a CGEvent with a retain count of 2 (why these aren't both 1 escapes me).

The C callback looks like this:

typedef CGEventRef __nullable (*CGEventTapCallBack)(CGEventTapProxy  proxy,
  CGEventType type, CGEventRef  event, void * __nullable userInfo);

This callback returns the supplied event or a newly-created one. In my code, I sometimes create a new event to return. In the C code, I did that with CGEventCreateCopy(), which gives me a new CGEventRef, and returned that.

In Swift, the callback signature looks like this:

public typealias CGEventTapCallBack = @convention(c) (CGEventTapProxy, CGEventType, CGEvent, UnsafeMutableRawPointer?) -> Unmanaged<CGEvent>?

and you make copies with

class CGEvent {
    public func copy() -> CGEvent?
}

Because the Swift callback returns an unmanaged reference, I was returning the event like this:

return Unmanaged.passUnretained(newEvent)

The callback has this comment:

The event passed to the callback is retained by the calling code, and is released after the callback returns and the data is passed back to the event system. If a different event is returned by the callback function,then that event will be released by the calling code along with the original event, after the event data has been passed back to the event system.

Maybe I should be returning the object using passRetained(), but that seems wrong to me, especially if I'm returning the event supplied to me (which is retained by the CGEvent system). It especially feels wrong that I should return it differently depending on whether or not I created a copy, and I don't understand why the C copy function has a higher retain count than the Swift copy function.

And I should note that if I do return it passRetained() without making a copy, I leak memory.

The documentation for passRetained and passUnretained talk about their use when passing to code: “This is useful when passing a reference to an API which Swift does not know the ownership rules for, but you know that the API expects you to pass the object at +0 [or +1 for passRetained].” But it doesn’t talk about returning to code that called you.

[tera]

I'm not 100% sure but IIRC the "pass" API's are useful when you are passing parameters and the "takeRetainedValue" / "takeUnretainedValue" API's are useful when returning results.

this.

[eskimo (Quinn “The Eskimo!”)]

why these aren't both 1 escapes me

Because the system has to have its own reference to the event so it can implement the “If a different event is returned” behaviour.

It especially feels wrong that I should return it differently
depending on whether or not I created a copy

Such is the nature of this API. Don’t get me wrong, it’s a weird beast, but it’s always been this way [1] and it’s obscure enough that I can see we didn’t got out of our way to fix it. And, yeah, it’s why CGEventTapCallBack is defined to return an unmanaged value.

First things first, the .copy method. It returns a managed reference, so Swift is guaranteed to release the +1 reference created by the underlying CGEventCreateCopy. That’s how Swift works in general.

With that in mind, there are two cases:

• You return the same event.

• You return a replacement event.

In the first case, the OS is holding a +1 reference to the event. It calls Swift and Swift is, in general, +0 on its incoming parameters. You are expected to return the original event without an extra reference, so you call passUnretained(_:) to get the unmanaged value you return. The OS maintains its responsibility for the original event’s +1 reference.

In the second case, the OS holding a +1 reference to the event. You call copy() which gives you a +0 reference. You’re supposed to return a +1 reference in this case, so you call passUnretained(_:) to get the unmanaged value to return. The OS notices that the new event is different from the old event, so it releases its +1 reference on the old event and takes responsibility for the +1 reference you returned.

‘Simple!’ (-:

Share and Enjoy

Quinn “The Eskimo!” @ DTS @ Apple

[1] I suspect that the CG folks were trying to avoid extra retain/release traffic.

[tera]

@eskimo

You call copy() which gives you a +0 reference.

How come?! create / copy make +1 references! I guess that was just a typo, as the rest still reads correctly :-)

[tera]

@JetForMe

After investigating, I discovered that CGEventCreateCopy() creates a CGEvent with a retain count of 3, while CGEvent.copy() creates a CGEvent with a retain count of 2 (why these aren't both 1 escapes me).

How do you check that? This gives 1 to me in both cases:

// swift
let event = CGEvent(keyboardEventSource: nil, virtualKey: 1, keyDown: true)!
print(myRetainCount(unsafeBitCast(event, to: Int.self))) // 1
let c = event.copy()!
print(myRetainCount(unsafeBitCast(c, to: Int.self))) // 1

// objc file with -fno-objc-arc on it:
int myRetainCount(long param) {
    return ((NSObject*)param).retainCount;
}

// bridging header:
int myRetainCount(long);

ditto this:

void testC(void) {
    CGEventRef event = CGEventCreateKeyboardEvent(NULL, 1, 1);
    printf("%d\n", [(NSObject*)event retainCount]); // 1
    CGEventRef event2 = CGEventCreateCopy(event);
    printf("%d\n", [(NSObject*)event2 retainCount]); // 1
}

[JetForMe (Rick M)]

@tera

How do you check that? This gives 1 to me in both cases:

I uploaded the project to Github. Running it from Xcode, you can see the counts in the Xcode console output.

I just realized I may be introducing a retain here.

@eskimo

Because the system has to have its own reference to the event so it can implement the “If a different event is returned” behaviour.

I’m talking about the copy I create inside my handler. I would’ve expected both copies (C or Swift) to come to me with a count of 1. The OS doesn’t know anything else about them, unless CGEventCreateCopy() has some global side-effect of inserting the event into some other retaining data structure.

Or am I introducing a retain because of the way I invoke the C method, via a C-interop wrapper (this line which calls this code).

@eskimo

First things first, the .copy method. It returns a managed reference, so Swift is guaranteed to release the +1 reference created by the underlying CGEventCreateCopy. That’s how Swift works in general.

I think this is the part I need to understand better. Does the Swift .copy() call CGEventCreateCopy() and then release once before giving it to me?

[tera]

@JetForMe

I just realized I may be introducing a retain here.

Yep. Let's try C code first:

void testC(void) {
    CGEventRef event = CGEventCreateKeyboardEvent(NULL, 1, 1);
    printf("%d\n", myRetainCount(event)); // 1
    printf("%d\n", CFGetRetainCount(event)); // 1
    CGEventRef event2 = CGEventCreateCopy(event);
    printf("%d\n", myRetainCount(event2)); // 1
    printf("%d\n", CFGetRetainCount(event2)); // 1
}

all good here, no surprises.

Using the following helpers on the C side:

int myRetainCount(long param) {
    return ((NSObject*)param).retainCount;
}

int myRetainCount2(long param) {
    return CFGetRetainCount((CFTypeRef)param);
}

int myRetainCount3(CFTypeRef param) {
    return CFGetRetainCount(param);
}

let's try it from swift:

    print(myRetainCount(unsafeBitCast(event, to: Int.self))) // 1
    print(myRetainCount2(unsafeBitCast(event, to: Int.self))) // 1

So far so good. Now:

    print(CFGetRetainCount(event)) // 2

Oops. Passing event inside CFGetRetainCount affects its retain count. But's that only temporary:

    print(myRetainCount2(unsafeBitCast(event, to: Int.self))) // 1, good again

Ditto:

    print(myRetainCount3(event)) // 2, oops
    print(myRetainCount2(unsafeBitCast(event, to: Int.self))) // 1, good again

By using the above "bit cast to int" trick I'm merely disabling all possible retain / release going under the hood when passing a ref-counted parameter inside a function, so it gives correct results.

@JetForMe

Does the Swift .copy() call CGEventCreateCopy() and then release once before giving it to me?

"event.copy" is CGEventCreateCopy:

    let c = event.copy()!
->  0x100006f1c <+1544>: bl     0x10008f0b4               ; symbol stub for: CGEventCreateCopy

It returns +1 object like all "create" and "copy" calls within Apple ecosystem.

[JetForMe (Rick M)]

So, my retain counts are all one greater than they actually are. That means the passed-in event is 3, the event I create by copying using C is 2, and the event I create by copying using Swift is 1.

I'm not sure why my C helper (that I call from Swift) to copy is increasing the retain count:

CGEventRef copyEvent(CGEventRef inEvent) {
	return CGEventCreateCopy(inEvent);
}

[tera]

@JetForMe

I'm not sure why my C helper (that I call from Swift) to copy is increasing the retain count:

It doesn't for me:

let c = copyEvent(event)
print(myRetainCount(unsafeBitCast(c, to: Int.self))) // 1

just don't do that to query retain count:

print(CFGetRetainCount(event)) // 2

ARC is hiding retain count logic from us so (among other things) it is free to do some fancy optimisations with those.

---

Were you able to fix the original problem?

Is this what you have?

    let newEvent = ... // +1 here, e.g. a copy
    return Unmanaged.passUnretained(newEvent)

which is equivalent to:

    let newEvent = ... // +1 here, e.g. a copy
    let result = Unmanaged.passUnretained(newEvent)
    // newEvent could be gone at this point as nothing holding it anymore
    return result // crash boom bang

that's the problem then. When creating a new event - return it at +1:

    let newEvent = ... // +1 here, e.g. a copy
    // nobody is holding newEvent, but us (via "newEvent" binding)
    return Unmanaged.passRetained(newEvent)
    // temporary +2, by the time of return +1 as it should be
    // (newEvent binding is going away so +2 will change to +1 as we return)

and when returning the original event:

    // the caller is holding originalEvent at some positive X
    // let's not change it
    return Unmanaged.passUnretained(originalEvent)

[JetForMe (Rick M)]

@tera

When creating a new event - return it at +1:

…

and when returning the original event:

…

Yeah, this is what I wanted to avoid; I had abstracted the actual handling of the event from the marshaling to and from C. I had to test if the event returned from my handler === the event passed in, and did that.

Then I realized I didn't need to make a copy of the event at all and just modified the incoming event to be what I wanted, sidestepping the entire problem of copying it.

Still, the Swift code has to do more work than equivalent C code would have to.

[eskimo (Quinn “The Eskimo!”)]

I think this is the part I need to understand better. Does the Swift
.copy() call CGEventCreateCopy

Yes.

and then release once before giving it to me?

No. If it did that, you wouldn’t be able to use it.

Rather, it calls CGEventCreateCopy, which returns a +1 reference, and then transfers the object to Swift’s managed reference ‘world’. At that point Swift is responsible for releasing the reference when you’re done with it.

Consider this code snippet:

@inline(never)
func test2(event: CGEvent) {
    print(event)
}

@inline(never)
func test(event: CGEvent) {
    let c = event.copy()
    test2(event: event)
}

test(…) disassembles like so:

(lldb) disas -f 
xxx`test(event:):
    0x100000e60 <+0>:  pushq  %rbp
    0x100000e61 <+1>:  movq   %rsp, %rbp
    0x100000e64 <+4>:  pushq  %r14
    0x100000e66 <+6>:  pushq  %rbx
    0x100000e67 <+7>:  movq   %rdi, %rbx
->  0x100000e6a <+10>: callq  0x100003589               ; symbol stub for: CGEventCreateCopy
    0x100000e6f <+15>: movq   %rax, %r14
    0x100000e72 <+18>: movq   %rbx, %rdi
    0x100000e75 <+21>: callq  0x100000de0               ; xxx.test2(event: __C.CGEventRef) -> () at main.swift:5
    0x100000e7a <+26>: movq   %r14, %rdi
    0x100000e7d <+29>: popq   %rbx
    0x100000e7e <+30>: popq   %r14
    0x100000e80 <+32>: popq   %rbp
    0x100000e81 <+33>: jmpq   *0x3181(%rip)             ; (void *)0x00007ff80d5d9c90: objc_release

<+15> saves the reference in r14. <+26> copies it to rdi, which then becomes the first argument to the objc_release call at <+33>.

Share and Enjoy

Quinn “The Eskimo!” @ DTS @ Apple

[tera]

@JetForMe

I had abstracted the actual handling of the event from the marshaling to and from C. I had to test if the event returned from my handler === the event passed in, and did that.

I see. That resulted into more code to do the abstraction and then yet more code to either passRetained or passUnretained to "undo" the abstraction when you go back to C.

@JetForMe

Then I realized I didn't need to make a copy of the event at all and just modified the incoming event to be what I wanted, sidestepping the entire problem of copying it.

So long as the caller is happy with that... I'd be somewhat cautious with this as the caller could be caught unprepared and could do (or could be revised to do):

CGEventRef newEvent = callback(event, ...);
if (newEvent != event)
    // do something important!
else
    // do nothing!

@JetForMe

Still, the Swift code has to do more work than equivalent C code would have to.

It is more due to Unmanaged:

CGEventRef callBack(CGEventTapProxy proxy, CGEventType type, CGEventRef originalEvent, void* userInfo) {
    if (someCondition) {
        return originalEvent;
    } else {
        return CGEventCreateKeyboardEvent(NULL, 1, 1);
    }
}

vs

func callBack(_ proxy: CGEventTapProxy, _ type: CGEventType, _ originalEvent: CGEvent, _ userInfo: UnsafeMutableRawPointer?) -> Unmanaged<CGEvent>? {
    if someCondition {
        return Unmanaged.passUnretained(originalEvent)
    } else {
        return Unmanaged.passRetained(CGEvent(keyboardEventSource: nil, virtualKey: 1, keyDown: 1))
    }
}

C API was written with manually managed memory in mind, so whatever ARC is doing you have to "undo" – hence a bit more code.