In a similar discussion here there was a post that somebody succeeded using SCRATCH instead of Alpine: Vapor 4 deployment on small docker image like for instance 12.1-alpine - #13 by rvsrvs . I don't know if that would be sufficient for you.
Another discussion on that topic from a year ago can be found here: Idea: runtime/slim images
regards,
Lars