Oooooh I get it now! Using -### was great, it actually explains a lot.
So it looks like a call with -output-file-map=... and -emit-silgen actually becomes several calls for individual files, in the form of
swift -frontend -emit-silgen foo.swift [...] -o foo.sil
So maybe I can implement support for something like
swift -frontend -dump-ast foo.swift [...] -o foo.ast
and then use the same -output-file-map trick to separate the initial call into one separate call per file.