Do you find any difference when Image is a struct?
I can only add to the chorus; Swift has a performance problem. It's often unpredictable and unexpected and I deal with it, as you did, through profiling. I also disable exclusivity checks for release (though I test with them on from time to time).
I'm optimistic about Swift performance. The recent ARC roadmap will go a long way, as will things like performance annotations in hot paths [1, 2]. It's unclear to me whether the ARC roadmap will improve exclusive memory access checks like these.