Access hasn't been lexically scoped since SE-0169, so "easier for the compiler" isn't really relevant here. It's really the other way around: files are a way for humans to organize code, and if you don't have separate files for "interface" and "implementation", then a very common place where you'll find code related to a particular declaration is in the same file. That's why fileprivate exists, and has existed since Swift 1.*
Moving forward from there, users made the argument that an even more common place to find uses is in the same scope (SE-0025). That was later widened to "the same type, in the same file" (SE-0169). That's private. Some people further argued that "in the same type but not in the same file" is also a useful grouping, and I disagree for the reason stated above.
So yeah, Swift's current access control is not flexible enough to do what you want, because there are two access levels that the language has to have, public and non-public, and we've split the latter into three groups we think are useful: internal, fileprivate, and private. They're not the only possible useful groups, just the ones most likely to be useful. Adding new levels, whether it's based on types, or file grouping, or declaration grouping like your example, increases the complexity of the language. Like any proposal, though, it might increase it in a way that's worth it.
And yeah, access control changes get a lot of pushback:
- There have been two changes to what
privatemeans since Swift 1, and they were both controversial. - Many people were not happy about
openbeing split frompublic. - File grouping is tied up with "submodules", which can mean a lot of different things. Proposing any of those features means thinking about whether the model can or should be expanded to include other features in the future, or whether they can coexist, and ugh.
- To my knowledge, no other mainstream language implements something like C++
friend. Which doesn't mean Swift can't, but does indicate that no one else has felt the need for it.
So the core team has a higher bar for access control changes, because they will take a lot of energy from the proposal author, from the core team themselves, and from the community.
This all comes back to header files. Header files offered no control over who used an API beyond public/non-public, but they did signal intent; clients had to opt in to seeing certain declarations. And since Swift doesn't have header files, that particular mechanism for opting in and signalling intent is lost as well.
P.S. I get that the problem is also exacerbated by certain declarations needing to be in the original type rather than in extensions. Maybe that's an angle that can be investigated as well.
* Full disclosure: I was a primary designer of Swift's original three-level access control, and still think that separating fileprivate from private was a mistake.