We implemented something similar to this: A Minimal Swift Docker Image. Off and on for over a year, I’ve been… | by Jeremy Jacobson | Medium, in which we basically start from a slimed down version of Ubuntu (abt 50MB) and add just the libraries the compiled Swift program links to. We are able to get images in the range of 150MB including our own code (NIO based server stuff).
Earlier, we had issues with Alpine Linux not being able to resolve hostname to ip from inside the container (see https://github.com/gliderlabs/docker-alpine/issues/8 which was not really resolved as of late last year). We haven't tried Alpine since.
cc: @ilyaK