Might this be the inappropriate actor stickiness in action, where the work hasn't properly detached from the main actor? Lowering the priority lets it work because the main actors native items are processed first?
@RonAvitzur If you can post an abstracted version of how and where you're enqueuing the work, perhaps we can see something.