Thank you!
I've seen this thread in the past but I was unable to find it.
I think this reply "Object was retained too many times" - #2 by Joe_Groff might be the solution :)
Edit: I can confirm this is exactly what I was searching for. Thank you again :) If anyone had anything to say about the topic (main why I should never try to do something like this :)) please let me know. From my perspective this seems as a reasonable thing to try. Slowing the Swift a bit is not a dealbreaker for me.