A Comprehensive Rethink of Access Levels in Swift

The name that has usually been used for this is `closed`. But no word that expresses a *restriction* is going to fit well into Swift’s access control system. The approach of Swift’s access control system (which I believe to be the best approach) is to express an upper bound on a *capability*. The basic capability is visibility of the symbol. That model is extended by talking about specific things a user might *do* with that symbol and setting an upper bound on that specific use: `internal private(set)`. `internal` is the bound on visibility and `private` is the bound on the ability to `set` the property.

`closed` is most naturally expressed in this system as `public internal(inherit)`. One interesting thing to observe is that `internal` is the default level of visibility. If we extend that default to capabilities as well as visibility the result is to give us exactly the behavior we want from a naked `public` annotation. We want to require an annotation for exposing anything (a symbol *or* a capability) outside the module. This means that `open` is really an alias for `public(inherit)` if we follow the principles underlying the model we have for access control.

Following this line of reasoning to its natural conclusion we can observe that the current interaction of `var` properties with access control *do not* follow the principles we have set forth for Swift. `public` variables *automatically* have `public` getters violating the principle that `internal` is the default. We could fix this by requiring users to say `public(set)`. I imagine there would be a lot of screaming in the short term if we proposed this. But it would put Swift’s access control system on a more consistent and principled footing.

I think there are three major reasons people have trouble with the current access control system. The `private` / `fileprivate` name situation is an obvious one. But I think perhaps more important is the fact that the system has idiosyncrasies such as introducing `open` rather than following the capability model of `set` (which would have resulted in `public(inherit)`). Another idiosyncrasy noted above is that `set` defaults to the level of the getter rather than `internal` (but bounded by visibility). Finally, the principle that underlies the access modifiers (modulo the idiosyncrasy of `open`) is that of a bounded scope. But this principle is implicit - each keyword is defined on an ad-hoc basis rather than making the underlying principle explicit and defining any shorthand we want in terms of that (as syntactic sugar for common cases and recommended defaults).

This is why I strongly believe the best approach is to make the underlying principle of scopes and capabilities with an `internal` default explicit. I think this principle is easy to teach and easy to understand once it is made explicit. It is a wonderful (but currently rather hidden) aspect of Swift. If we can teach users to understand this principle and to learn the names of scopes and capabilities they will not find it difficult and complex. They will find it as easy to learn as a library function: we would have `scope(<capbability name>, <scope name>)`**. The capability would be defaulted to basic visibility. The scope would be defaulted to the current scope.

Consider scope(inherit, public)`. It would read like “the scope of inheritance is public”. It tells the reader *exactly* what is happening. We could conceptualize shortened like `open` as something like `accessalias open = scope(inherit, public)`. Tools could even allow you to cmd-Click to the definition to see a definition like this. Users would learn something that is no more complicated than a single function call and any time they are confused by shorthand they could cmd-Click or look up documentation that explains it clearly in terms of this "function call”. Note: `accessalias` wouldn’t be a real thing, just a way to document and explain the shorthand keywords we have for “soft defaults” and common conveniences.

This seems to me like a very simple and elegant yet extraordinarily powerful system. I think it’s much better than a small collection of ad-hoc definitions of very specific behaviors.

**Anyone who has been reading my thoughts on this might notice that I am using `scope` rather than `scoped` in this post. As I was writing this I realized that when we think of access in terms of capability it reads better with `scope` than it does with `scoped` (i.e. “inheritance is scoped public” does not read as well as “the scope of inheritance is public".

The nice thing about that is that we could declare something as private public(inherit), making it not callable but possible to override, which would be a great fit for methods that are only intended as override points and are not meant to be called directly, like the many such methods on NSView.

That's interesting. I hadn't thought about this. It would complicate the model a little bit - I have been thinking in terms of capabilities as additive to the basic capability (i.e. when a setter is available so is the getter, when you can override you can also call, etc).

That said, there have been requests for override-but-not-call and you're right that this can be an important guarantee for a library. Would you expect override to imply the ability to call super? Or would you expect this contract to be used in cases where the subclass is not allowed to call super at all? That latter obviously fits the semantics of this access control model better and I imagine it is often what you would want (for example if the override points are part of a template method pattern).

There have also been requests for set-only properties although I'm not as familiar with the use cases for that.

···

Sent from my iPad

On Feb 26, 2017, at 10:58 AM, Charles Srstka <cocoadev@charlessoft.com> wrote:

On Feb 25, 2017, at 5:27 PM, Matthew Johnson via swift-evolution <swift-evolution@swift.org> wrote:

Charles

Just joking of course (???)

Exposure levels: World, Module, File, Type (= [ W | M | F | T ])
Access control: Read, Write, Override, Inheritable (= [ R | W | O | I ])

let api = Access(w, [r, -w, -o, -i]) // Access must always be statically resolved
let mod = Access(m, [r, w, -o, i])
let cust = Access(api, mod, (f, [r, -w, o, i]), (t, [r, w, o, i]))

access(cust) var a: Int

Ok, joking besides…

I think the above suggestion is madness. Sure it would work and it is not even close to the worst possible solution.

In effect what I see people suggesting is exactly the above, only differently phrased:

fileprivate = access((w, [-r, -w, -o, -i]), (m, [-r, -w, -o, -i]), (f, [r, w, o, i]), (t, [r, w, o, i])
public private(set) = access((w, [r, -w, -o, -i]), (m, [r, -w, -o, -i]), (f, [r, -w, -o , i]), (t, [r, w, o, i])
open = …
final = ...

etc, you get the drift
What I am trying to show in this example is the complexity of trying to implement the full set of possible controls by introducing keywords (or keyword combinations!) for each possible combination.

I like simplicity. The TO suggestion would suit me just fine.

But there is a tip over point: when we get too many keywords then it becomes simpler to dump the keyword approach and go for the full Monty.

IMO in Swift 3 we are already dangerously close to the tip-over point.

Regards,
Rien

Site: http://balancingrock.nl
Blog: http://swiftrien.blogspot.com
Github: Balancingrock (Rien) · GitHub
Project: http://swiftfire.nl

+1000

This is the best of the proposals I have seen. I think it works by itself as a complete proposal, but since we are talking "comprehensive rethink", there is one use case which most of the proposals seem to miss.

That is, what if I want to make a class which is both Public and Subclassable, but I want to keep some of the implementation details private to the class, while still allowing subclasses and extensions (across the module boundary) to have access to those details. A concrete example of this is UIGestureRecognizer, which is intended to be subclassed outside of its framework, but hides potentially catastrophic things (like being able to set the state) from callers. This isn’t currently possible in Swift AFAICT without importing from ObjC.

My solution to this has been to allow something to be marked ‘public hidden’ or ‘hidden', which means that it is public/internal, but has it’s visibility limited only to files which opt-in to seeing it with ‘import hidden filename’ (instead of just ‘import filename’). I think this is the simplest way to provide this feature, which is sometimes very necessary. It also captures authorial intent very nicely.

My submodule proposal is able to support this use case. You would just put the symbols intended for subclasses in a separate submodule. These submodules could be exposed to users with any name you desire, regardless of the internal structure of your module. You could even arrange for users to get all of the non-subclass-only symbols automatically when they import your module but require explicit import of each individual subclass-only submodule in places where it is needed.

I agree that this could also be solved with nested submodules

A minor note, but you would not need to nest them. You could do it either way.

I can’t think of how to do this, but I trust you that it can.

, but the current proposals all seem to add a lot of complexity. This complexity does give you much finer grain control over visibility, but what I like about Nevin’s proposal is that it is ‘just enough’ control with minimal complexity.

Determining what “just enough” is is obviously a very subjective thing. :) Everyone seems to have their own opinion about this (more than usual) which is going to make it one of the more complicated discussions we’ll have on the list.

I worked very hard on my proposal to try to design a system that stays out of your way until you need the tools it offers. That way nobody is faced with complexity unless they are deriving direct benefit from it, but when they need powerful tools those tools are available. This is the idea of progressive disclosure (as I understand it).

True. I would personally define “just enough” as easy for the 80% and possible for the 98%.

I don’t think your proposal is bad at all. I just like the simplicity of Nevin’s a little more (especially with the confusion around ‘open’). In fact, I would like to see you combine them a bit (see below).

What I like about ‘hidden’ vs export of nested submodules is that you can freely intermix those declarations in your code (like you can with private now).

Yeah, this is an advantage. It’s not a bad idea, but it’s not a replacement for submodules either. They are complementary.

Agreed.

What I would really like to see is Nevin’s proposal with a simplified version of yours providing the submodules, and something like ‘hidden’ replacing the idea of ‘export’.

Are you saying you want to see all submodules exposed externally automatically? Or are you only talking about non-top-level export?

In more detail, I really like just having ‘private’, ‘internal’, and ‘public’ where private means private to the submodule. It is very simple and understandable. I think having a single level of submodule using your ‘submodule’ syntax is the simplest way to provide those.

I think there is going to be *a lot* of resistance to the idea of removing file-level private. Most people want to keep that, but just revert than name to `private` like we had in Swift 2.

I think I would be ok with nested submodules as well, again using your syntax, as long as we have ‘private’ mean private within the current submodule (and children). If we find we really do need parametrized private (or equivalent) to provide visibility to a specific parent scope, that is something that can be an additive (non-breaking) change later. I think we will be able do pretty much everything we need to without it though.

I could be wrong, but I’d be willing to bet that in a submodule world you would have a very vocal and large number of people complaining if we suggest not including any of: module-wide visibility, submodule-wide visibility and file-wide visibility. There are certainly people who think that one or another of these is not worth supporting but I don’t see anything near a consensus around avoiding any one of these. Each will have a very vocal group of proponents that I believe (again I could be wrong) would be a majority with respect to that particular scope.

I can understand why somebody might prefer a style that avoids relying one of these scopes (people have their idiosyncrasies after all). What I have a hard time understanding is why somebody would propose that *nobody* be able to scope a symbol to one of these very physical and prominent scopes.

One thing `hidden` doesn’t do is allow you to say *why* the symbols are hidden and make that clear when they are imported. I can imagine allowing it to take a parameter that is a user-defined identifier stating *why* the symbols are hidden. For example, I might have `hidden(subclass)` and `hidden(extension)`. Making it a user-defined identifier would encourage names that mean something, require users to have read the documentation telling them what the identifier is, and make it clear to readers of the code why hidden symbols are imported (if the identifier used does not match any available symbols it would result in a compiler error).

A user-defined identifier would also be better than a small set of language-defined arguments because users would expect language-defined identifiers like “subclass” or “extension” to be *enforced* by the language. There has been strong pushback against doing that for several reasons. A parameterized `hidden` is a general feature that could express what is desired very well while not over-promising and under-delivering `protected` or `typeprivate` are extremely permeable restrictions and therefore don’t offer any real guarantees that `hidden` or a factored out submodule wouldn’t offer. It’s better to have a more general feature that does exactly what it says and leads users to believe it does.

I disagree. The whole point of ‘hidden’ is to protect the *callers* of a type from accidentally invoking its deliberate extension points. It basically means “don’t touch these unless you understand them”, and having to ‘import hidden’ is explicitly taking responsibility for this. I don’t think making extenders type a secret code would actually help anything, and would just annoy/complicate. There is only so much you can do to protect people from themselves.

Right now, anything that would be marked ‘public hidden’ has to be marked ‘public’, so I think it is a thorough improvement over the status quo.

That’s a reasonable perspective. This feature seems like it could certainly be useful in some cases but it’s applicable in relatively narrow cases and is orthogonal to the things I am interested in focusing on right now so I don’t plan to propose it. I’m happy to wait and see if somebody else decides to run with it.

···

On Feb 26, 2017, at 10:47 PM, Jonathan Hull <jhull@gbis.com> wrote:

On Feb 25, 2017, at 2:41 PM, Matthew Johnson <matthew@anandabits.com <mailto:matthew@anandabits.com>> wrote:

On Feb 25, 2017, at 4:01 PM, Jonathan Hull <jhull@gbis.com <mailto:jhull@gbis.com>> wrote:

On Feb 25, 2017, at 1:19 PM, Matthew Johnson <matthew@anandabits.com <mailto:matthew@anandabits.com>> wrote:

On Feb 25, 2017, at 2:54 PM, Jonathan Hull via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

There is also the question of how to separate a single type into multiple submodules, when we can’t declare storage in an extension.

We have discussed allowing storage to be declared in an extension in the past on the list. There is reasonable support for adding this eventually and we have ideas about how to do it (the complexity is primarily around insuring proper initialization of storage declared in extensions). Introducing a submodules feature could well drive demand for it by demonstrating concrete use cases where it becomes more necessary.

Even without that feature it is possible to separate a type into separate submodules using the features in my proposal. The details of how you do that would depend on concrete use cases.

True, and I do want this feature… but I don’t want to rely on it being there when it is not yet on the roadmap.

Thanks,
Jon

What I like about ‘hidden’ vs export of nested submodules is that you can freely intermix those declarations in your code (like you can with private now).

Yeah, this is an advantage. It’s not a bad idea, but it’s not a replacement for submodules either. They are complementary.

Agreed.

What I would really like to see is Nevin’s proposal with a simplified version of yours providing the submodules, and something like ‘hidden’ replacing the idea of ‘export’.

Are you saying you want to see all submodules exposed externally automatically? Or are you only talking about non-top-level export?

In Nevin’s system (to the best of my understanding), Submodules don’t really have much to do with visibility except to define a boundary. Things marked ‘private' are visible within a submodule, but not outside of it. Things marked ‘internal’ are visible within the entire module, but not outside of it. Things marked ‘public’ are visible everywhere.

Submodules don’t have visibility of their own in this system (which is part of what I like about it). Once you start having to worry about the visibility of the submodule itself, it really complicates the model, and you find yourself thinking on a much more abstract level instead of writing code.

In more detail, I really like just having ‘private’, ‘internal’, and ‘public’ where private means private to the submodule. It is very simple and understandable. I think having a single level of submodule using your ‘submodule’ syntax is the simplest way to provide those.

I think there is going to be *a lot* of resistance to the idea of removing file-level private. Most people want to keep that, but just revert than name to `private` like we had in Swift 2.

As I said above, Nevin’s system redefines ‘private’ to always mean private to the enclosing submodule. If you have not defined an explicit submodule for your file, then the file itself is implicitly it’s own submodule (aka. the same behavior as fileprivate)

I think I would be ok with nested submodules as well, again using your syntax, as long as we have ‘private’ mean private within the current submodule (and children). If we find we really do need parametrized private (or equivalent) to provide visibility to a specific parent scope, that is something that can be an additive (non-breaking) change later. I think we will be able do pretty much everything we need to without it though.

I could be wrong, but I’d be willing to bet that in a submodule world you would have a very vocal and large number of people complaining if we suggest not including any of: module-wide visibility, submodule-wide visibility and file-wide visibility.

These are all provided for under the proposal (with file-wide visibility for anonymous submodules).

If we wanted to allow nested submodules though, ‘private’ would still mean private to your containing submodule. Other things in the submodule could see it, as could any child submodules, but anything outside the submodule could not. What I am saying is that I don’t think we will need 'private(submoduleName)’, but we can always add it in a non-breaking way if I am wrong about that.

There are certainly people who think that one or another of these is not worth supporting but I don’t see anything near a consensus around avoiding any one of these. Each will have a very vocal group of proponents that I believe (again I could be wrong) would be a majority with respect to that particular scope.

I can understand why somebody might prefer a style that avoids relying one of these scopes (people have their idiosyncrasies after all). What I have a hard time understanding is why somebody would propose that *nobody* be able to scope a symbol to one of these very physical and prominent scopes.

The only scope which would be lost is private to the type (what we currently call ‘private’ in Swift 3). Basically it works exactly like Swift 2 did, but you have the added ability to extend the domain of private by using submodules…

I like your notation for defining submodules, and would like to see that added, as well as my idea of ‘hidden’ to cover the needs I talked about before.

Thanks,
Jon

Ah, the irony…

Having argued against too much complexity, I now encountered an error in my software that shows why there might indeed be a need for “open” and “public”.

In an API:

class Foo {
  open func foo(options: Option ...) {
    foo(options)
  }
  open func foo(options: [Options]) { … }
}

An API user might do this:

class Bar: Foo {
  override func foo(options: Option ...) {
    …
    super.foo(options)
  }
}

var b = Bar()
b.foo(options: [Option.thisOne])

This compiles fine, and will even work .... up to a point where the additions in Bar.foo are essential. Then it fails.
The user should have called: b.foo(options: Option.thisOne)

This can easily be prevented by making Foo.foo(options: Option) public instead of open.
Then the user would not have been able to override the convenience method.
Furthermore: the user would not have experienced any inconvenience either as Foo.foo() would still be usable.

The interesting thing is that in this case there is no real argument on the side of the API developer to have a need for either open or public. It is entirely to the benefit of the API user. To me that makes a case for having both open and public. As an API developer I am willing to “go the extra mile” to create a good user experience, even if that means using tricks to achieve that end. However that would be impossible in this case. This end-user convenience is only achievable through open and public.

Regards,
Rien

Site: http://balancingrock.nl
Blog: http://swiftrien.blogspot.com
Github: Balancingrock (Rien) · GitHub
Project: http://swiftfire.nl

Rien, your example shows two things (well, three, but one relates to
variadic arguments not access levels).

First, it shows the importance of being able to prevent external overrides
of individual methods. I did not directly address that in my pitch, however
it could easily use the same syntax as closed classes, namely
‘final(public)’. I am not tied to that particular spelling, it just seemed
logical—better ideas are welcome.

Second, your example shows that even with today’s “soft default” of
closedness, developers still sometimes erroneously allow subclassing or
overriding. To put that another way, when a library author thinks something
should be ‘open’ in Swift 3, they mark it as such. The decision to make a
class or method closed is an intentional one, even today. Having it be a
“soft default” does not and cannot prevent the type of error it is alleged
to.

Also, unless there are subclasses in your library which override that
variadic convenience method, it should probably be marked ‘final’ and just
call through to the overrideable workhorse array-based version. No ‘closed’
required here. This provides further evidence that most of the time,
methods should either be overrideable or final, as ‘closed’ is a rare niche
case.

Nevin

as ‘closed’ is a rare niche case.

If there is agreement on the importance of closed, imho we shouldn't even need variants of final:
You could already build the required class hierarchies if the restriction that a subclass can't have higher visibility as its parent is lifted.

Imho this should be done anyways, and then you could simply define an internal baseclass with "public final" subclasses.