Using file system folders as a visibility grouping mechanism feels like something you would need to go "all or nothing" on to not confuse users; for example, how Java package names must match the folder structure of the sources being compiled. That's not the case for modules in Swift today, so it would introduce some artificial restrictions; for example, in use cases involving generated code, build systems often generate code outside of the source tree, and code that is conceptually in the same module wouldn't be able to use that visibility level. Maybe that specific use case is an acceptable sacrifice, but it shows the sensitivity of introducing these kinds of non-lexical boundaries.
The way around that would be to have compiler arguments that let you define the groups explicitly, so you could say "these source files belong to this group, regardless of their location", but at that point you're essentially defining something that parallels submodules.
My own personal opinion is that the time folks want to spend/are spending auditing and fine-tuning the access control on their own code is far greater than the actual benefit they derive from it.