Good question! Swift's binary "swiftmodule" format is already a lot faster and lazier than reading from source, so the need for an even more optimized format hasn't come up as much. The one thing that's using more indirection than the Clang serialized AST format is cross-module references, which are basically using a constrained form of lookup instead of some offset into the other module structure. This means they don't need to be rebuilt (in theory) if the dependency changes in some compatible way.
(On the other hand, no one has quite the volume of Swift code to compare to the amount of Objective-C that's processed when you do #import <UIKit/UIKit.h>, so maybe in the future we'll want to build additional "fast cross-module references".)
The parseable interfaces that serve as the stable module format do have a speed problem, and so we do compile those to a binary format (the existing one) and stash them in the same module cache location that Clang uses.