[Draft] scope-based submodules

I didn't expect submodules to be a part of the Swift 4 discussion. When it came up I was pleasantly surprised. I have been thinking about the design of a submodule system for quite a while but was planning to wait until it was clearly in scope to draft a proposal. Now that the topic has been introduced I decide to write down the design I've been thinking about, along with the motivation and goals that underly it. I understand submodules may not be in scope for Swift 4 but wanted to contribute this design while the discussion is fresh in everyone's mind.

I am including the contents of the proposal below. You can also find it on Github: swift-evolution/NNNN-scope-based-submodules.md at scope-based-submodules · anandabits/swift-evolution · GitHub

I am very much looking forward to everyone's feedback!

Matthew
Scope-based submodules

Proposal: SE-NNNN <file:///Users/Matthew/Dropbox/Matthew/Development/__notes/__swift/evolution/NNNN-scope-based-submodules.md>
Authors: Matthew Johnson <https://github.com/anandabits&gt;
Review Manager: TBD
Status: Awaiting review
Introduction

This proposal describes a submodule system based on the principle of strictly nested scopes. The design strives to follow the Swift philosophy of offering good defaults while progressively disclosing very powerful tools that can be used by experts to solve complex problems.

Motivation

Swift currently provides two kinds of entities that provide system boundaries without directly introducing a symbol: modules and files*.

Modules introduce an ABI boundary, a name boundary, and a scope boundary.
Files introduce a scope boundary that carries no additional semantics.
Swift currently lacks the ability to introduce a name and scope boundary without also introducing an ABI boundary. Such a boundary would be naturally situated halfway between in terms of strength and ceremony. The lack of such a boundary significantly limits our abiltiy to structure a large Swift program. Introducing a way to form this kind of boundary will provide a powerful tool for giving internal structure to a module.

*The important aspect of a file in Swift is the logical scope boundary it introduces. The physical file system representation is incidental to this. The appendix on file system independence discusses this in more detail.

Goals

Any discussion of submodules inevitably reveals that there are very different perspectives of what a submodule is, what problems submodules should be able to solve, etc. This section describes the design goals of this proposal in order to facilitate evaluation of both the goals themselves as well as how well the solution accomplishes those goals.

The primary goal of this proposal are to introduce a unit of encapsulation within a module that is larger than a file as a means of adding explicit structure to a large program. All other goals are subordinate to this goal and should be considered in light of it.

Some other goals of this proposal are:

Submodules should help us to manage and understand the internal dependencies of a large, complex system.
Submodules should be able to collaborate with peer submodules without necessarily being exposed to the rest of the module.
A module should not be required to expose its internal submodule structure to users when symbols are exported.
It should be possible to extract a submodule from existing code with minimal friction. The only difficulty should be breaking any circular dependencies.
Some additional non-functional requirements for the solution are:

Submodules should not negatively impact runtime performance. WMO should be able to see across submodule boundaries.
Submodules should not negatively impact build performance. Ideally they will improve build performance by giving the compiler more high-level information about internal dependencies.
Deferred goal:

It is not an immediate goal to support submodules in single-file scripts. The appendix discussing file system independence discusses some ideas that could be used to support single-file scripts in the future.
Proposed solution

There are several relatively orthogonal aspects to the design of a submodule system. A design must answer the following questions:

How is code placed in a submodule?
How are symbols in one submodule made available to another submodule or module?
How do submodules interact with access control?
This proposal diverages a little bit from the usual proposal format to faciliate discussion of alternatives within the context of each aspect of the chosen design. In each case an alternative could be substituted without compromising the overall design.

Note: This proposal uses the term “top-level submodule” to distinguish the scope of code that is not explicitly placed in a submodule from the module as a whole. The top-level submodule is in most respects identical to any other parent submodule. There are two differences: 1) it is the only submodule that can export symbols outside of the module and 2) any open or public symbols it declares are automatically exported according to their access modifier.

Placing code in a submodule

Each file is part of the top level submodule by default.

A submodule declaration may be used at the top of a file:

Only the first non-comment, non-whitespace line may contain a submodule declaration.
The submodule decalaration looks like this: submodule MySubmoduleName
A submodule declaration may not be prefixed with the module name.
Submodule names form a hierarchical path:

The fully qualified name of the submodule specified by Submodule.InnerSubmodule is: MyModuleName.Submodule.InnerSubmodule.
In this example, InnerSubmodule is a child of Submodule.
A submodule may not have the same name as any of its ancestors. This follows the rule used by types.
Submodules may not be extended. They form strictly nested scopes.

The only way to place code in a submodule is with a submodule declaration at the top of a file.
All code in a file exists in a single submodule.
A module is made up of strictly nested scoped that look like this:

The hierarchy of nested scopes in scope-based submodules
Alternatives

Grouping mechanisms

There are several other ways to specify which submodule the top-level scope of a file is in. All of these alternatives share a crucial problem: you can’t tell what submodule your code is in by looking at the file.

The alternatives are:

Use a manifest file. This would be painful to maintain.
Use file system paths. This is too tightly coupled to physical organization. Appendix A discusses file system independence in more detail.
Leave this up to the build system. This makes it more difficult for a module to support multiple build systems.
Require all files to include a submodule declaration

We could require all files to include an explicit submodule declaration. However, that would be a breaking change and would violate the principle of progressive disclosure. Users should not need to know about submodules if they don’t use them.

Allow submodule references to explicitly state the name of the module

The module name is implicit throughout the scope of the entire module. Specifying it explicitly is redundant. Prohibiting explicit mention of the module name offers more flexibility for combining submodules into build products.

Visibility of submodules

The export statement is used to modifiy the visibiltiy of a submodule within the module. It is also used by the top-level module to publish submodules to clients of the module.

All submodules are implicitly exported with module-wide visibility by default (and hidden outside of the module by default*).
All submodules are implicitly available for export outside the module.
A submodule may use an explicit export statement to modify the visibility of a descendent submodule.
export statements are only allowed at the top level of a file.
*The exception to this is that open and public symbols in the top-level submodule are always exported exactly as declared.

Top-level export

All export statements consist of an access modifier, the export keyword, and a submodule name:

open export ChildSubmodule

When this export statement appears in the top-level submodule, ChildSubmodule becomes available for import by clients of the submodule with the fully qualified name Module.ChildSubmodule. public exports are also available. When public is used all published symbols in the exported submodule have a maximum visiblity of publicregardless of how they were declared.

Top-level public and open export statement may be modified with the following options:

A submodule may be published under a different external name using the export as NewName syntax*.
@implicit causes symbols from the submodule to be implicitly imported when the module is imported.
@inline causes the symbols from the submodule to appear as if they had been declared directly within the top-level submodule.
@implicit may be combined with renaming but @inline may not appear along with either of them.
*When a submodule is renamed for export with the as clause its internal name does not change. A submodule always has the same fully qualfied name everywhere within its module.

Here are some example export statements:

// All symbols in `Child1` are available for import by clients as `Module.Foo`
// These symbols are *not* imported automatically when a client imports the module with `import Module`
public export Child1 as Foo

// All symbols in `Child2` are available for explicit import by clients as `Module.Child2`
// The symbols are also automatically imported when a client imports the module with `import Module`
// When the symbols are imported implicitly they retain the fully qualified name prefix of `Module.Child2`
@implicit open export Child2

// All symbols in `Child3` are available for explicit import by clients as `Module.Foo.Bar`
// The symbols are also automatically imported when a client imports the module with `import Module`
// When the symbols are imported implicitly they retain the fully qualified name prefix of `Module.Foo.Bar`
@implicit open export Child3 as Foo.Bar

// All symbols in `Child4.Grandchild` appear to clients as if they had been declared
// directly in the top-level submodule.
// If the process of inlining the symbols produces duplicate symbols a compiler error is produced
// at the site of one or both of the `export` statements.
@inline public export Child4.Grandchild

// All symbols in `Child5.Grandchild` are available for explicit import by clients as `Module.Foo`
// along with the symbols declared in `Child`.
// The symbols are also automatically imported when a client imports the module with `import Module`
// When the symbols are imported implicitly they retain the fully qualified name prefix of `Module.Foo`
// As with `@inline`, when two submodules are given the same external name a duplicate symbol error may occur.
@implicit public export Child5.Grandchild as Foo
One interesting observation is that both Child1 and Child5.Grandchild are renamed to Foo. The symbols declared former is not implicitly imported by import Module but the latter is, despite having the same fully qualified name prefix.

Exports within the module

A submodule may bound the maximum visibility of any of its descendent submodules by explicitly exporting it:

// `Child1.Grandchild` may be exported by the top-level module, but only with `public` visibility.
// `Child1.Grandchild` may not be exported as `open`.
public export Child1.Grandchild

// `Child2` is exported with `internal` visibility.
// Because `internal` is scoped to the submodule level *only* the parent submodule can see `Child2`.
// No submodules except the direct parent of `Child2` (the current submodule) are allowed to `import Child2`.
// This also implies that the `Child2` may not be exported to clients because the top-level
// submodule is not able to see or reference `Child2` at all.
internal export Child2
The access modifier may specify a scope internal or greater.
Only the direct parent of a submodule may specify the internal modifier. A grandparent cannot hide a grandchild from its parent.
If a descendent includes an export statement for the same submodule, the access modifier must be no greater than the access modifier specified by the descendent. An ancestor may provide a tighter bound to visibility but may not increase visibility. An attempt to increase visibility results in an error.
Note: If a submodule is not visible none of its descendents is visible either.

Alternatives

Use import access modifiers to export a submodule

The semantics of placing a bound on the visibility of a descendent submodule is significantly different than the semantics of importing symbols from a submodule into the current lexical scope. Mixing the semantics of the two is confusing.

Restrict the visibility of a submodule to its parent unless the parent explicitly exports it.

Users should be able to use submodules without needing to place export statements in every parent submodule. Module-wide default visibility for submodules is analagous to internal default visibility for symbols.

Require all submodules to be visible module-wide.

This removes an important tool for bounded collaboration within a complex system. A parent submodule should be allowed to have a child submodule(s) which are implementation details of the parent and not exposed to the rest of the module.

Allow renaming to be used by export statements within the module.

A submodule should have the same fully qualified name everywhere it is used within a single module, whether that be the declaring module or a client module. The declaring module and client modules may see different names, but each sees a name that is consistent fully qualified name everywhere the submodule is referenced.

Allow @inline to be used by export statements within the module.

As with renaming, a symbol should have a single fully qualified name everywhere within a single module.

Allow @implicit to be used by export statements within the module.

This would reduce the visibility of internal dependencies. If we find that import-per-submodule becomes boilerplate-y this is an easy feature to add later.

Importing submodules

Submodules are imported in exactly the same way as an external module by using an import statement. There are a few additional details that are not applicable for external modules:

Circular imports are not allowed.
A submodule may not import any of its ancestors.
Relative child and sibling names are allowed using the same rules that apply to nested types.
Access control

An access modifier applies an upper bound to the scope in which a symbol or submodule is visible. With the introduction of submodules, internal now applies at the level of a submodule: only the current submodule may see an internal entity. We need a new way to specify module-wide visibility.

This proposal builds on Option 2 in the proposal Fix Private Access Levels which reverts private to the Swift 2 meaning (equivalent to fileprivate) and uses scoped for the Swift 3 scoped access feature. It does this by allowing the scoped access modifier to be parameterized with a scope reference. By defaults it references the scope in which it appears, but any ancestor scope may be specified as a parameter.

The paremeterization of the scoped access modifier provides a simple yet powerful way for a submodule to bound the visibility of a descendent.

Some examples of using scoped exports are:

submodule Parent

// `Grandparent` and all of its descendents can see `Child1` (fully qualified: `Grandparent.Parent.Child1`)
// This reads: `Child1` is scoped to `Grandparent`.
scoped(Grandparent) export Child1

// `Child2` is visible throughout the module but may not be exported for use by clients.
// This reads: `Child2` is scoped to the module.
scoped(module) export Child2
With parameterization, scoped has the power to specify all access levels that Swift has today:

`scoped` == `private` (Swift 3)
`scoped(file)` == `private` (Swift 2 & 4?) == `fileprivate` (Swift 3)
`scoped(submodule)` == `internal`
`scoped(public) scoped(internal, inherit)`* == `public`
`scoped(public)` == `open`
This design is a direct generalization of the principle underlying Swift’s existing access control system. It unifies the semantics of the system under the single elegant mechanism of ancestor scope references.

While it is possible to specify all access levels using scoped that is not recommended. The aliases public, private(Swift 2) and internal provide excellent default access levels that don’t require a user to think about scope hierarchies. Using the default access levels when possible calls extra attention to cases where a different choice was made.

*This is a conceptual model. This proposal does not introduce the inherit or override parameter to access modifiers. It could be added in the future as a way to bound inheritance within a module. It would work similarly to private(set) does in Swift today.

Aside

The parameterization of scoped also allows us to reference other scopes that we cannot in today’s system, specifically extensions: scoped(extension) and outer types: scoped(TypeName).

Alternatives

If we don’t adopt the approach of parameterizing scoped our options for access control include:

Submodules are only allowed to see public and open symbols from other submodules

A module-wide scope is highly desirable. People might avoid using submodules if this is not available.

This approach also creates a lot more friction when refactoring. A possible workaround to the lack of a module-wide scope in this system is to place code in a non-exported submodule and declare symbols public. Even with the workaround, extracting a submodule may not always be possible or desirable and the public access modifiers required would be misleading. It would be much better to be able to state our intent directly.

Use internal to cover the whole module and private to cover a submodule

One suggestion that has appeared is the idea of removing fileprivate and making private be submodule-wide. internal would remain module-wide. This is too coarse - many people want a file-level scope.

internal is Swift’s default access modifier. A symbol with default access modifier should not be able to cross a submodule boundary implicitly.

Add the moduleinternal access modifier

This is about as ugly as fileprivate.

Detailed design

Export errors

Multiple exports of the same submodule

If a submodule exports the same descendent more than once and the semantics of the declarations are not identical an error is produced.

Symbol flattening

When a submodule is exported by the top-level module using the @inline attribute it is possible that there will be conflicting symbol definitions in the child and the top-level submodule (or other inlined submodules). This results in a compiler error at the site of the conflicing @inline export statements.

Overlapping renames

As with flattening, when two or more submodules are given the same external name symbol conflicts are possible. This also results in a compiler error at the site of the conflicting export as statements.

Access errors during export if the specified access modifier exceeds maximum

An error is produced when an export statement includes an access modifier greater than the bound provided for the exported submodule by a descendent of the exporting submodule.

Source compatibility

This proposal is purely additive. That said, it would be a breaking change for the standard library to move existing API into an externally visible submodule.

Effect on ABI stability

This proposal is purely additive. That said, it would be a breaking change for the standard library to move existing API into an externally visible submodule.

Effect on API resilience

This proposal is purely additive.

Future directions

Selective export and import

The ability to import and export individual symbols would be a very nice to have.

Scoped import

The ability to import modules, submodules, and symbols into any lexical scope would be nice to have.

Bounded inheritance

It could be useful to have the ability to bound inheritance within a module. This could be accomplished by inroducing inherit and override parameters for access modifiers (which would work similarly to the existing set parameter).

Appendix A: file system independence

The submodule design specified by this proposal is file system independent. The only relationship it has with the physical file system is that a file introduces an anonymous scope boundary which is referenced by scoped(file) or fileprivate (Swift 3) or private (Swift 2 and 4?).

The logical role of a “file” in this design is to provide a boundary of encapsulation that is even lighter weight than a submodule: it doesn’t hide names. All declarations are implicitly available not only within the file but also across the file boundary (modulo access control). Files are to submodules as submodules are to modules.

If a future version of Swift were to eliminate files in favor of some kind of code browser it would still be very useful to have the ability to form a pure scope boundary (with no additional semantics). A scope declaration could be used to do this. Scope declarations could have an optional (or required) name and could even be nested. In this system privatewould reference the nearest anonymous ancestor scope declaration (or would be removed if we don’t allow anonymous scopes).

The logical structure of this design can be directly translated into a grammar that could be represented directly with syntax. Such a grammer could be used to support scripts with submodules. An example follows:

// A module contans a single implicit, anonymous submodule.
// submodule {
  // A submodule may contain `scope` declarations (i.e. files) as well as other submodules.
  // An anonymous scope is equivalent to a file in current Swift.
  // If we introduce lexical scopes we would probably require them to be named explicitly.
  // This example uses the anonymous scope in order to most closely match the role files play in the current system.
  // Because `scope` does not provide a name boundary all names declared in one scope
  // are visible in other scopes (modulo access control)
  scope {
      // Top-level declarations go here.
      // This is equivalent to the top level of a file in Swift today.
      // It is also equivalent to the top level of a file that does not contain
      // a `submodule` declaration in this proposal.
      
      // It would be possible to allow nested, named scopes.
      // A scope name participates in the scope and name hierachies.
      // However, it does not form a name boundary like a submodule does.
      scope Named {
        // This declares the static variable `Named.foo`
        // `scoped(file)` references the nearest anonymous ancestor scope.
        // It is used in this example for specificity.
        // Real code would use the alias `private` or `fileprivate`
        // If we introduce explicit scope syntax we would probably want a better name to refer
        // to the nearest anonymous scope than `file` or we may just require all scopes to have a name.
        scoped(file) var foo: String
      }
      // `Named.foo` is visible here
  }
  // `Named.foo` is not visible here.
  
  submodule Foo {}
  submodule Baz {}
  submodule Buzz {
    // Equivalient to a file in current Swift.
    scope {
      // submodule declarations go here.
      // This is equivalent to the top level scope of a file that contains the `submodule Foo` declaration.
    }
    scope {}
    submodule Baz {}
  }
//}
Appendix B: namespace style submodules

It is possible to design a system that allows a name boundary to be formed without also forming a scope boundary. A natural consequence of this is that symbols may be placed into a namespace-style submodule in many (unlimited) scopes via extension (even extension outside the module is theoretically possible). Allowing this runs contray to both of the two primary goals of this proposal (encapsulation and structure).

Allowing a submodule to be extended in multiple scopes precludes the possibility of submodule internal visibility. A submodule internal access modifier could still be defined but it would not provide the guarantee it purports to. The submodule can be opened by extension anywhere within the module. If a lazy developer wants to access a submodule internal symbol from a distant subsytem all they need to do is add an extension and wrap the submodule internal symbol with a new symbol offering higher visibility*. In such a system there is the same wide gap between file scope and module scope that exists today.

Allowing a submodule to be extended in multiple scopes precludes the ability to introduce real structure to a module. We are able to introduce structure to the names but not the module itself. The structure of a submodule in such a system may be widely dispersed throughout the module. It is not part of a strictly hierarchical structure of scopes which each having a single designated location within the larger structure.

What you do get from name boundaries that do not also form a scope boundary is a soft form of symbol hiding (soft because all submodules are available for import or extension anywhere within the program). This does provide some value, but not nearly as much value as is provided by a name boundary that is accompanied by a scope boundary.

Another downside to namespace-style submodules that are open to extension is that they are much less likely to facilitate improved build performance because they don’t add any physical structure to the system.

Finally, lexical submodules have the potential to be confusing. If submodules form a name boundary (even a soft one) an import statement is required to access the symbols declared inside a submodule. Is code that surrounds a lexical submodule declaration able to see the symbols it declares without importing them? Most developers will expect the symbols to be available. It is probably necessary to make an exception to name boundary for the surrounding lexical context. However, if an exception is made then this system relies heavily on a file to provide a bound to the implicit symbol import.

*It is worth observing that the ability to violate encapsulation via extension (or subclassing) is one of the primary reasons Swift does not offer type-based access modifiers such as typeprivate or protected. The do not offer true encapsulation at all. They are a statement of intent that cannot really be verified in the way that is desired. They form a permeable rather than a hard boundary.

I didn't expect submodules to be a part of the Swift 4 discussion. When it came up I was pleasantly surprised. I have been thinking about the design of a submodule system for quite a while but was planning to wait until it was clearly in scope to draft a proposal. Now that the topic has been introduced I decide to write down the design I've been thinking about, along with the motivation and goals that underly it. I understand submodules may not be in scope for Swift 4 but wanted to contribute this design while the discussion is fresh in everyone's mind.

I am including the contents of the proposal below. You can also find it on Github: https://github.com/anandabits/swift-evolution/blob/scope-based-submodules/proposals/NNNN-scope-based-submodules.md

I am very much looking forward to everyone's feedback!

Matthew
Scope-based submodules

Proposal: SE-NNNN <file:///Users/Matthew/Dropbox/Matthew/Development/__notes/__swift/evolution/NNNN-scope-based-submodules.md>
Authors: Matthew Johnson <https://github.com/anandabits&gt;
Review Manager: TBD
Status: Awaiting review
Introduction

This proposal describes a submodule system based on the principle of strictly nested scopes. The design strives to follow the Swift philosophy of offering good defaults while progressively disclosing very powerful tools that can be used by experts to solve complex problems.

Motivation

Swift currently provides two kinds of entities that provide system boundaries without directly introducing a symbol: modules and files*.

Modules introduce an ABI boundary, a name boundary, and a scope boundary.
Files introduce a scope boundary that carries no additional semantics.
Swift currently lacks the ability to introduce a name and scope boundary without also introducing an ABI boundary. Such a boundary would be naturally situated halfway between in terms of strength and ceremony. The lack of such a boundary significantly limits our abiltiy to structure a large Swift program. Introducing a way to form this kind of boundary will provide a powerful tool for giving internal structure to a module.

*The important aspect of a file in Swift is the logical scope boundary it introduces. The physical file system representation is incidental to this. The appendix on file system independence discusses this in more detail.

Goals

Any discussion of submodules inevitably reveals that there are very different perspectives of what a submodule is, what problems submodules should be able to solve, etc. This section describes the design goals of this proposal in order to facilitate evaluation of both the goals themselves as well as how well the solution accomplishes those goals.

The primary goal of this proposal are to introduce a unit of encapsulation within a module that is larger than a file as a means of adding explicit structure to a large program. All other goals are subordinate to this goal and should be considered in light of it.

Some other goals of this proposal are:

Submodules should help us to manage and understand the internal dependencies of a large, complex system.
Submodules should be able to collaborate with peer submodules without necessarily being exposed to the rest of the module.
A module should not be required to expose its internal submodule structure to users when symbols are exported.
It should be possible to extract a submodule from existing code with minimal friction. The only difficulty should be breaking any circular dependencies.
Some additional non-functional requirements for the solution are:

Submodules should not negatively impact runtime performance. WMO should be able to see across submodule boundaries.
Submodules should not negatively impact build performance. Ideally they will improve build performance by giving the compiler more high-level information about internal dependencies.
Deferred goal:

It is not an immediate goal to support submodules in single-file scripts. The appendix discussing file system independence discusses some ideas that could be used to support single-file scripts in the future.
Proposed solution

There are several relatively orthogonal aspects to the design of a submodule system. A design must answer the following questions:

How is code placed in a submodule?
How are symbols in one submodule made available to another submodule or module?
How do submodules interact with access control?
This proposal diverages a little bit from the usual proposal format to faciliate discussion of alternatives within the context of each aspect of the chosen design. In each case an alternative could be substituted without compromising the overall design.

Note: This proposal uses the term “top-level submodule” to distinguish the scope of code that is not explicitly placed in a submodule from the module as a whole. The top-level submodule is in most respects identical to any other parent submodule. There are two differences: 1) it is the only submodule that can export symbols outside of the module and 2) any open or public symbols it declares are automatically exported according to their access modifier.

Placing code in a submodule

Each file is part of the top level submodule by default.

A submodule declaration may be used at the top of a file:

Only the first non-comment, non-whitespace line may contain a submodule declaration.
The submodule decalaration looks like this: submodule MySubmoduleName
A submodule declaration may not be prefixed with the module name.
Submodule names form a hierarchical path:

The fully qualified name of the submodule specified by Submodule.InnerSubmodule is: MyModuleName.Submodule.InnerSubmodule.
In this example, InnerSubmodule is a child of Submodule.
A submodule may not have the same name as any of its ancestors. This follows the rule used by types.
Submodules may not be extended. They form strictly nested scopes.

The only way to place code in a submodule is with a submodule declaration at the top of a file.
All code in a file exists in a single submodule.
A module is made up of strictly nested scoped that look like this:

<scope-based-submodules.png>
The hierarchy of nested scopes in scope-based submodules
Alternatives

Grouping mechanisms

There are several other ways to specify which submodule the top-level scope of a file is in. All of these alternatives share a crucial problem: you can’t tell what submodule your code is in by looking at the file.

The alternatives are:

Use a manifest file. This would be painful to maintain.
Use file system paths. This is too tightly coupled to physical organization. Appendix A discusses file system independence in more detail.
Leave this up to the build system. This makes it more difficult for a module to support multiple build systems.
Require all files to include a submodule declaration

We could require all files to include an explicit submodule declaration. However, that would be a breaking change and would violate the principle of progressive disclosure. Users should not need to know about submodules if they don’t use them.

Allow submodule references to explicitly state the name of the module

The module name is implicit throughout the scope of the entire module. Specifying it explicitly is redundant. Prohibiting explicit mention of the module name offers more flexibility for combining submodules into build products.

Visibility of submodules

The export statement is used to modifiy the visibiltiy of a submodule within the module. It is also used by the top-level module to publish submodules to clients of the module.

All submodules are implicitly exported with module-wide visibility by default (and hidden outside of the module by default*).
All submodules are implicitly available for export outside the module.
A submodule may use an explicit export statement to modify the visibility of a descendent submodule.

I don’t see the need for modifying the visibility of descendent modules. Can you give me an idea why its important enough to warrant all that extra syntax?

export statements are only allowed at the top level of a file.
*The exception to this is that open and public symbols in the top-level submodule are always exported exactly as declared.

Top-level export

All export statements consist of an access modifier, the export keyword, and a submodule name:

open export ChildSubmodule

When this export statement appears in the top-level submodule, ChildSubmodule becomes available for import by clients of the submodule with the fully qualified name Module.ChildSubmodule. public exports are also available. When public is used all published symbols in the exported submodule have a maximum visiblity of publicregardless of how they were declared.

Top-level public and open export statement may be modified with the following options:

A submodule may be published under a different external name using the export as NewName syntax*.
@implicit causes symbols from the submodule to be implicitly imported when the module is imported.
@inline causes the symbols from the submodule to appear as if they had been declared directly within the top-level submodule.
@implicit may be combined with renaming but @inline may not appear along with either of them.
*When a submodule is renamed for export with the as clause its internal name does not change. A submodule always has the same fully qualfied name everywhere within its module.

Here are some example export statements:

// All symbols in `Child1` are available for import by clients as `Module.Foo`
// These symbols are *not* imported automatically when a client imports the module with `import Module`
public export Child1 as Foo

// All symbols in `Child2` are available for explicit import by clients as `Module.Child2`
// The symbols are also automatically imported when a client imports the module with `import Module`
// When the symbols are imported implicitly they retain the fully qualified name prefix of `Module.Child2`
@implicit open export Child2

// All symbols in `Child3` are available for explicit import by clients as `Module.Foo.Bar`
// The symbols are also automatically imported when a client imports the module with `import Module`
// When the symbols are imported implicitly they retain the fully qualified name prefix of `Module.Foo.Bar`
@implicit open export Child3 as Foo.Bar

// All symbols in `Child4.Grandchild` appear to clients as if they had been declared
// directly in the top-level submodule.
// If the process of inlining the symbols produces duplicate symbols a compiler error is produced
// at the site of one or both of the `export` statements.
@inline public export Child4.Grandchild

// All symbols in `Child5.Grandchild` are available for explicit import by clients as `Module.Foo`
// along with the symbols declared in `Child`.
// The symbols are also automatically imported when a client imports the module with `import Module`
// When the symbols are imported implicitly they retain the fully qualified name prefix of `Module.Foo`
// As with `@inline`, when two submodules are given the same external name a duplicate symbol error may occur.
@implicit public export Child5.Grandchild as Foo
One interesting observation is that both Child1 and Child5.Grandchild are renamed to Foo. The symbols declared former is not implicitly imported by import Module but the latter is, despite having the same fully qualified name prefix.

Exports within the module

A submodule may bound the maximum visibility of any of its descendent submodules by explicitly exporting it:

// `Child1.Grandchild` may be exported by the top-level module, but only with `public` visibility.
// `Child1.Grandchild` may not be exported as `open`.
public export Child1.Grandchild

// `Child2` is exported with `internal` visibility.
// Because `internal` is scoped to the submodule level *only* the parent submodule can see `Child2`.
// No submodules except the direct parent of `Child2` (the current submodule) are allowed to `import Child2`.
// This also implies that the `Child2` may not be exported to clients because the top-level
// submodule is not able to see or reference `Child2` at all.
internal export Child2
The access modifier may specify a scope internal or greater.
Only the direct parent of a submodule may specify the internal modifier. A grandparent cannot hide a grandchild from its parent.
If a descendent includes an export statement for the same submodule, the access modifier must be no greater than the access modifier specified by the descendent. An ancestor may provide a tighter bound to visibility but may not increase visibility. An attempt to increase visibility results in an error.
Note: If a submodule is not visible none of its descendents is visible either.

Alternatives

Use import access modifiers to export a submodule

The semantics of placing a bound on the visibility of a descendent submodule is significantly different than the semantics of importing symbols from a submodule into the current lexical scope. Mixing the semantics of the two is confusing.

Restrict the visibility of a submodule to its parent unless the parent explicitly exports it.

Users should be able to use submodules without needing to place export statements in every parent submodule. Module-wide default visibility for submodules is analagous to internal default visibility for symbols.

Require all submodules to be visible module-wide.

This removes an important tool for bounded collaboration within a complex system. A parent submodule should be allowed to have a child submodule(s) which are implementation details of the parent and not exposed to the rest of the module.

IMHO, the implementation details of a submodule should live in that submodule, not in a child submodule. Shouldn’t the main purpose of submodules be to help organise publicly available code into name-spaces, and not add to add scoping mechanisms for implementation?

For example, looking at C#’s System.Xml library, the namespaces clearly help to separate code, like serialisation classes from more mundane Xml reading classes, etc...

System.Xml
System.Xml.Linq
System.Xml.Resolvers
System.Xml.Schema
System.Xml.Serialization
System.Xml.Serialization.Advanced
System.Xml.Serialization.Configuration
System.Xml.XmlConfiguration
System.Xml.XPath
System.Xml.Xsl
System.Xml.Xsl.Runtime

Allow renaming to be used by export statements within the module.

A submodule should have the same fully qualified name everywhere it is used within a single module, whether that be the declaring module or a client module. The declaring module and client modules may see different names, but each sees a name that is consistent fully qualified name everywhere the submodule is referenced.

Allow @inline to be used by export statements within the module.

As with renaming, a symbol should have a single fully qualified name everywhere within a single module.

Allow @implicit to be used by export statements within the module.

This would reduce the visibility of internal dependencies. If we find that import-per-submodule becomes boilerplate-y this is an easy feature to add later.

Importing submodules

Submodules are imported in exactly the same way as an external module by using an import statement. There are a few additional details that are not applicable for external modules:

Circular imports are not allowed.
A submodule may not import any of its ancestors.
Relative child and sibling names are allowed using the same rules that apply to nested types.
Access control

An access modifier applies an upper bound to the scope in which a symbol or submodule is visible. With the introduction of submodules, internal now applies at the level of a submodule: only the current submodule may see an internal entity. We need a new way to specify module-wide visibility.

This proposal builds on Option 2 in the proposal Fix Private Access Levels which reverts private to the Swift 2 meaning (equivalent to fileprivate) and uses scoped for the Swift 3 scoped access feature. It does this by allowing the scoped access modifier to be parameterized with a scope reference. By defaults it references the scope in which it appears, but any ancestor scope may be specified as a parameter.

The paremeterization of the scoped access modifier provides a simple yet powerful way for a submodule to bound the visibility of a descendent.

Some examples of using scoped exports are:

submodule Parent

// `Grandparent` and all of its descendents can see `Child1` (fully qualified: `Grandparent.Parent.Child1`)
// This reads: `Child1` is scoped to `Grandparent`.
scoped(Grandparent) export Child1

// `Child2` is visible throughout the module but may not be exported for use by clients.
// This reads: `Child2` is scoped to the module.
scoped(module) export Child2
With parameterization, scoped has the power to specify all access levels that Swift has today:

`scoped` == `private` (Swift 3)
`scoped(file)` == `private` (Swift 2 & 4?) == `fileprivate` (Swift 3)
`scoped(submodule)` == `internal`
`scoped(public) scoped(internal, inherit)`* == `public`
`scoped(public)` == `open`
This design is a direct generalization of the principle underlying Swift’s existing access control system. It unifies the semantics of the system under the single elegant mechanism of ancestor scope references.

While it is possible to specify all access levels using scoped that is not recommended. The aliases public, private(Swift 2) and internal provide excellent default access levels that don’t require a user to think about scope hierarchies. Using the default access levels when possible calls extra attention to cases where a different choice was made.

*This is a conceptual model. This proposal does not introduce the inherit or override parameter to access modifiers. It could be added in the future as a way to bound inheritance within a module. It would work similarly to private(set) does in Swift today.

Like we’ve had time to discuss, I’m really not sold on this solution as it adds a fair amount of complexity, and complexity is exactly what people have been complaining about in the current model. The complexity I’m not sure we need:

new keyword
more than 1 way to express the same semantics
the parametrisation of the keyword

Aside

The parameterization of scoped also allows us to reference other scopes that we cannot in today’s system, specifically extensions: scoped(extension) and outer types: scoped(TypeName).

Alternatives

If we don’t adopt the approach of parameterizing scoped our options for access control include:

Submodules are only allowed to see public and open symbols from other submodules

A module-wide scope is highly desirable. People might avoid using submodules if this is not available.

This approach also creates a lot more friction when refactoring. A possible workaround to the lack of a module-wide scope in this system is to place code in a non-exported submodule and declare symbols public. Even with the workaround, extracting a submodule may not always be possible or desirable and the public access modifiers required would be misleading. It would be much better to be able to state our intent directly.

Use internal to cover the whole module and private to cover a submodule

One suggestion that has appeared is the idea of removing fileprivate and making private be submodule-wide. internal would remain module-wide. This is too coarse - many people want a file-level scope.

I agree that this is a bad idea.

internal is Swift’s default access modifier. A symbol with default access modifier should not be able to cross a submodule boundary implicitly.

Add the moduleinternal access modifier

This is about as ugly as fileprivate.

What about the alternative of not having a submodule scope at all? Submodules would see all internal access in all submodules, as long as it’s imported at the top of the file.

I think the divide comes from the fact that I ant “namespaces” as a name grouping mechanism and you want “submodules” as a scope grouping mechanism.

···

On 24 Feb 2017, at 20:34, Matthew Johnson via swift-evolution <swift-evolution@swift.org> wrote:
Detailed design

Export errors

Multiple exports of the same submodule

If a submodule exports the same descendent more than once and the semantics of the declarations are not identical an error is produced.

Symbol flattening

When a submodule is exported by the top-level module using the @inline attribute it is possible that there will be conflicting symbol definitions in the child and the top-level submodule (or other inlined submodules). This results in a compiler error at the site of the conflicing @inline export statements.

Overlapping renames

As with flattening, when two or more submodules are given the same external name symbol conflicts are possible. This also results in a compiler error at the site of the conflicting export as statements.

Access errors during export if the specified access modifier exceeds maximum

An error is produced when an export statement includes an access modifier greater than the bound provided for the exported submodule by a descendent of the exporting submodule.

Source compatibility

This proposal is purely additive. That said, it would be a breaking change for the standard library to move existing API into an externally visible submodule.

Effect on ABI stability

This proposal is purely additive. That said, it would be a breaking change for the standard library to move existing API into an externally visible submodule.

Effect on API resilience

This proposal is purely additive.

Future directions

Selective export and import

The ability to import and export individual symbols would be a very nice to have.

Scoped import

The ability to import modules, submodules, and symbols into any lexical scope would be nice to have.

Bounded inheritance

It could be useful to have the ability to bound inheritance within a module. This could be accomplished by inroducing inherit and override parameters for access modifiers (which would work similarly to the existing set parameter).

Appendix A: file system independence

The submodule design specified by this proposal is file system independent. The only relationship it has with the physical file system is that a file introduces an anonymous scope boundary which is referenced by scoped(file) or fileprivate (Swift 3) or private (Swift 2 and 4?).

The logical role of a “file” in this design is to provide a boundary of encapsulation that is even lighter weight than a submodule: it doesn’t hide names. All declarations are implicitly available not only within the file but also across the file boundary (modulo access control). Files are to submodules as submodules are to modules.

If a future version of Swift were to eliminate files in favor of some kind of code browser it would still be very useful to have the ability to form a pure scope boundary (with no additional semantics). A scope declaration could be used to do this. Scope declarations could have an optional (or required) name and could even be nested. In this system privatewould reference the nearest anonymous ancestor scope declaration (or would be removed if we don’t allow anonymous scopes).

The logical structure of this design can be directly translated into a grammar that could be represented directly with syntax. Such a grammer could be used to support scripts with submodules. An example follows:

// A module contans a single implicit, anonymous submodule.
// submodule {
  // A submodule may contain `scope` declarations (i.e. files) as well as other submodules.
  // An anonymous scope is equivalent to a file in current Swift.
  // If we introduce lexical scopes we would probably require them to be named explicitly.
  // This example uses the anonymous scope in order to most closely match the role files play in the current system.
  // Because `scope` does not provide a name boundary all names declared in one scope
  // are visible in other scopes (modulo access control)
  scope {
      // Top-level declarations go here.
      // This is equivalent to the top level of a file in Swift today.
      // It is also equivalent to the top level of a file that does not contain
      // a `submodule` declaration in this proposal.
      
      // It would be possible to allow nested, named scopes.
      // A scope name participates in the scope and name hierachies.
      // However, it does not form a name boundary like a submodule does.
      scope Named {
        // This declares the static variable `Named.foo`
        // `scoped(file)` references the nearest anonymous ancestor scope.
        // It is used in this example for specificity.
        // Real code would use the alias `private` or `fileprivate`
        // If we introduce explicit scope syntax we would probably want a better name to refer
        // to the nearest anonymous scope than `file` or we may just require all scopes to have a name.
        scoped(file) var foo: String
      }
      // `Named.foo` is visible here
  }
  // `Named.foo` is not visible here.
  
  submodule Foo {}
  submodule Baz {}
  submodule Buzz {
    // Equivalient to a file in current Swift.
    scope {
      // submodule declarations go here.
      // This is equivalent to the top level scope of a file that contains the `submodule Foo` declaration.
    }
    scope {}
    submodule Baz {}
  }
//}
Appendix B: namespace style submodules

It is possible to design a system that allows a name boundary to be formed without also forming a scope boundary. A natural consequence of this is that symbols may be placed into a namespace-style submodule in many (unlimited) scopes via extension (even extension outside the module is theoretically possible). Allowing this runs contray to both of the two primary goals of this proposal (encapsulation and structure).

Allowing a submodule to be extended in multiple scopes precludes the possibility of submodule internal visibility. A submodule internal access modifier could still be defined but it would not provide the guarantee it purports to. The submodule can be opened by extension anywhere within the module. If a lazy developer wants to access a submodule internal symbol from a distant subsytem all they need to do is add an extension and wrap the submodule internal symbol with a new symbol offering higher visibility*. In such a system there is the same wide gap between file scope and module scope that exists today.

Allowing a submodule to be extended in multiple scopes precludes the ability to introduce real structure to a module. We are able to introduce structure to the names but not the module itself. The structure of a submodule in such a system may be widely dispersed throughout the module. It is not part of a strictly hierarchical structure of scopes which each having a single designated location within the larger structure.

What you do get from name boundaries that do not also form a scope boundary is a soft form of symbol hiding (soft because all submodules are available for import or extension anywhere within the program). This does provide some value, but not nearly as much value as is provided by a name boundary that is accompanied by a scope boundary.

Another downside to namespace-style submodules that are open to extension is that they are much less likely to facilitate improved build performance because they don’t add any physical structure to the system.

Finally, lexical submodules have the potential to be confusing. If submodules form a name boundary (even a soft one) an import statement is required to access the symbols declared inside a submodule. Is code that surrounds a lexical submodule declaration able to see the symbols it declares without importing them? Most developers will expect the symbols to be available. It is probably necessary to make an exception to name boundary for the surrounding lexical context. However, if an exception is made then this system relies heavily on a file to provide a bound to the implicit symbol import.

*It is worth observing that the ability to violate encapsulation via extension (or subclassing) is one of the primary reasons Swift does not offer type-based access modifiers such as typeprivate or protected. The do not offer true encapsulation at all. They are a statement of intent that cannot really be verified in the way that is desired. They form a permeable rather than a hard boundary.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Matthew, thank you for sharing this great text!
Unfortunately was not able to read it in details yet, but after looking for proposed solution, have a quick question.

You are proposing to have 'submodule' keyword in each file of that submodule. How can we then solve such issue: imagine that I have swift file with some code in it. I got this file from external source(repository/site), I will periodically update it with newer version for example. I want to use this file in my two submodules, but don't want to 'share' or 'make visible' types/classes/funcs etc from this file with other code outside of these submodules.

As I understand, with your suggestion the only way is have two copies of that file and append 'submodule' keyword in each. Or I understand the whole idea incorrectly? (Sorry then)

···

On 24.02.2017 22:34, Matthew Johnson via swift-evolution wrote:

I didn't expect submodules to be a part of the Swift 4 discussion. When it
came up I was pleasantly surprised. I have been thinking about the design
of a submodule system for quite a while but was planning to wait until it
was clearly in scope to draft a proposal. Now that the topic has been
introduced I decide to write down the design I've been thinking about,
along with the motivation and goals that underly it. I understand
submodules may not be in scope for Swift 4 but wanted to contribute this
design while the discussion is fresh in everyone's mind.

I am including the contents of the proposal below. You can also find it on
Github: https://github.com/anandabits/swift-evolution/blob/scope-based-submodules/proposals/NNNN-scope-based-submodules.md

I am very much looking forward to everyone's feedback!
.....

Scope-based submodules

  • Proposal: SE-NNNN
  • Authors: Matthew Johnson
  • Review Manager: TBD
  • Status: Awaiting review

Well, this is certainly comprehensive! Sorry about the delay in answering; I've been hosting a house guest and haven't had a lot of free time.

The primary goal of this proposal are to introduce a unit of encapsulation within a module that is larger than a file as a means of adding explicit structure to a large program. All other goals are subordinate to this goal and should be considered in light of it.

I agree with this as the primary goal of a submodule system.

Some other goals of this proposal are:

  • Submodules should help us to manage and understand the internal dependencies of a large, complex system.
  • Submodules should be able to collaborate with peer submodules without necessarily being exposed to the rest of the module.
  • A module should not be required to expose its internal submodule structure to users when symbols are exported.
  • It should be possible to extract a submodule from existing code with minimal friction. The only difficulty should be breaking any circular dependencies.

One goal I don't see mentioned here is "segment the API surface exposed to importing code". The `UIGestureRecognizerSubclass` use case has been thoroughly discussed, but I think there are probably a lot of cases where there are two "sides" to an API and it'd often be helpful to hide one unless it's needed. `URLProtocol` and `URLProtocolClient` come to mind; the many weird little classes and symbols related to `NSAtomicStore` and `NSIncrementalStore` might be another.

(I'm not necessarily suggesting that the Foundation and Core Data overlays should move these into submodules—I'm suggesting that, if they were implemented in a Swift with a submodule feature, they would be candidates for submodule encapsulation.)

Submodule names form a hierarchical path:

  • The fully qualified name of the submodule specified by Submodule.InnerSubmodule is: MyModuleName.Submodule.InnerSubmodule.
  • In this example, InnerSubmodule is a child of Submodule.
  • A submodule may not have the same name as any of its ancestors. This follows the rule used by types.

Does being in a nested submodule have any semantic effect, or is it just a naming trick?

Submodules may not be extended. They form strictly nested scopes.

  • The only way to place code in a submodule is with a submodule declaration at the top of a file.
  • All code in a file exists in a single submodule.

I'm a big supporter of the 1-to-N submodule-to-file approach.

There are several other ways to specify which submodule the top-level scope of a file is in. All of these alternatives share a crucial problem: you can’t tell what submodule your code is in by looking at the file.

The alternatives are:

  • Use a manifest file. This would be painful to maintain.
  • Use file system paths. This is too tightly coupled to physical organization. Appendix A discusses file system independence in more detail.
  • Leave this up to the build system. This makes it more difficult for a module to support multiple build systems.

I'm going to push back on this a little. I don't like the top-of-file `submodule` declaration for several reasons:

  1. Declarations in a Swift file are almost always order-independent. There certainly aren't any that must be the first thing in the file to be valid.

  2. Swift usually keeps configuration stuff out of source files so you can copy and paste snippets of code or whole files around with minimum fuss. Putting `submodule` declarations in files means that developers would need to open and modify those files if they wanted to copy them to a different project. (It's worth noting that your own goal of making it easy to extract submodules into separate modules is undermined by submodule declarations inside files.)

  3. However you're organizing your source code—whether in the file system, an IDE project, or whatever else—it's very likely that you will end up organizing files by submodule. That means either information about submodules will have to be specified twice—once in a canonical declaration and again in source file organization—and kept in sync, or IDEs and tooling will have to interpret the `submodule` declarations in source files and reflect that information in their UIs.

  4. Your cited reason for rejecting build system-based approaches is that "This makes it more difficult for a module to support multiple build systems", but Swift has this same problem in *many* other parts of its design. For instance, module names and dependencies are build system concerns, despite the fact that this makes it harder to support multiple build systems. I can only conclude that supporting multiple build systems with a single code base is, in the long term, a non-goal, presumably by improving the Xcode/SwiftPM story in some way.

I'm still a fan of build-system-based approaches because I think they're better about these issues. The only way that they're worse is that—as you note—it may not be clear which submodule a particular file is in. But I think this is basically a UI problem for editors.

Top-level export

All export statements consist of an access modifier, the export keyword, and a submodule name:

open export ChildSubmodule

Is the access control keyword mandatory?

If we do indeed use `submodule` statements, could we attach the attributes to them, rather than having a separate statement in a different file?

What does it mean if a `public` or `open` symbol is in a submodule which is not `export`ed?

  • A submodule may be published under a different external name using the export as NewName syntax*.

What's the use case for this feature?

  • @implicit causes symbols from the submodule to be implicitly imported when the module is imported.
  • @inline causes the symbols from the submodule to appear as if they had been declared directly within the top-level submodule.

So if you write `@implicit public export Bar` in module `Foo`, then writing `import Foo` also imports `Foo.Bar.Baz` *as* `Foo.Bar.Baz`, whereas `@inline public export Bar` copies `Foo.Bar.Baz` into `Foo`, so it imports as `Foo.Baz`?

What's the use case for supporting both of these behaviors?

Exports within the module

A submodule may bound the maximum visibility of any of its descendent submodules by explicitly exporting it:

I'm not sure how valuable this feature is in this kind of submodule design.

···

On Feb 24, 2017, at 11:34 AM, Matthew Johnson via swift-evolution <swift-evolution@swift.org> wrote:

***

To avoid being coy, here's the export control model that *I* think would make the most sense for this general class of submodule system designs:

1. A submodule with `public` or `open` symbols is importable from outside the module. There is no need to separately mark the submodule as importable.

2. Normally, references within the module to submodule symbols need to be prefixed with the submodule name. (That is, in top-level `Foo` code, you need to write `Bar.Baz` to access `Foo.Bar.Baz`). As a convenience, you can import a submodule, which makes the submodule's symbols available to that file as though they were top-level module symbols.

3. When you import a submodule, you can mark it with `@exported`; this indicates that the symbols in that submodule should be aliased and, if `public` or `open`, re-exported to other modules.

4. There are no special facilities for renaming submodules or implicitly importing submodules.

Importing submodules

Submodules are imported in exactly the same way as an external module by using an import statement.

Okay, but what exactly does importing *do*? Set up un-prefixed private aliases for the submodule's internal-and-up APIs?

There are a few additional details that are not applicable for external modules:

  • Circular imports are not allowed.

Why not? In this design, all submodules are evaluated at once, so I'm not sure why circular imports would be a problem.

// `Grandparent` and all of its descendents can see `Child1` (fully qualified: `Grandparent.Parent.Child1`)
// This reads: `Child1` is scoped to `Grandparent`.

scoped(Grandparent) export Child1

// `Child2` is visible throughout the module but may not be exported for use by clients.
// This reads: `Child2` is scoped to the module.

scoped(module) export Child2

With parameterization, scoped has the power to specify all access levels that Swift has today:

`scoped` == `private` (Swift 3)
`scoped(file)` == `private` (Swift 2 & 4?) == `fileprivate` (Swift 3)
`scoped(submodule)` == `internal`
`scoped(public) scoped(internal, inherit)`* == `public`
`scoped(public)` == `open`

The parameterization of scoped also allows us to reference other scopes that we cannot in today’s system, specifically extensions: scoped(extension) and outer types: scoped(TypeName).

What is the purpose of creating more verbose aliases for existing access levels? I can't think of one, which means that these are redundant.

And if we remove them as redundant, the remaining access control levels look like:

  scoped
  private
  scoped(TypeName)
  internal
  scoped(SomeModule)
  scoped(module)
  scoped(extension)
  public
  open

There's just no logic to the use of the `scoped` keyword here—it doesn't really mean anything other than "we didn't want to assign a keyword to this access level".

***

I think we need to go back to first principles here. The reason to introduce a new access level is that we believe that a submodule is a large enough unit of code that it will simultaneously need to encapsulate some of its implementation details from other submodules, *and* have some of its own implementation details encapsulated from the rest of the submodule. Thus, we need at least three access levels within a submodule: one that exposes an API to other submodules, one that exposes an API throughout a submodule, and one that exposes it to only part of a submodule.

What we do *not* need is a way to allow access only from certain other named submodules. The goal is to separate external and internal interfaces, not to micromanage who can access what.

Basically, that means we need one of two things. Keeping all existing keywords the same—i.e., not removing either `private` or `fileprivate`— and using `semi-` as a placeholder, we want to either have:

  private: surrounding scope
  fileprivate: surrounding file
  semi-internal: surrounding submodule
  internal: surrounding module
  public: all modules (no subclassing)
  open: all modules (with subclassing)

Or:

  private: surrounding scope
  fileprivate: surrounding file
  internal: surrounding submodule
  semi-public: surrounding module
  public: all modules (no subclassing)
  open: all modules (with subclassing)

The difference between the two is that, with `semi-internal` below `internal`, submodule APIs are exposed by default to other submodules; with `semi-public` above `internal`, submodule APIs are encapsulated by default from other submodules.

I think encapsulating by default is the right decision, so we want the `semi-public` design. But there's also a second reason to use that design: We can anticipate another use case for it. The library resilience design document discusses the idea of "resilience domains"—groups of libraries whose versions are always matched, and which therefore don't need to use resilient representations of each others' data structures—and the idea of having "SPIs", basically APIs that are only public to certain clients. I think these ideas could be conflated, so that a semi-public API would be available both to other submodules in the module and to other libraries in your resilience domain, and that this feature could be used to expose SPIs.

So, that leaves an important question: what the hell do you call this thing? My best suggestions are `confidential` and `privileged`; in the context of information, these are both used to describe information which *is* shared, but only within a select group. (Think, for instance, of attorney-client privilege: You can share this information with your lawyer, but not with anyone else.)

So in short, I suggest adding a single access level to the existing system:

  private
  fileprivate
  internal
  confidential/privileged
  public
  open

This is orthogonal to any other simplification of the access control system, like removing `private` or `fileprivate`.

Appendix A: file system independence

I think we need to decide: Is a translation unit of some sort—whether it's a physical on-disk file or some simulacrum like a database record or just a separate string—something intrinsic to Swift? I think it should be; it simplifies a lot of parts of the language that would otherwise require nesting and explicit scoping.

If translation units are an implicit part of Swift, then this section is not really necessary. If translation units aren't, then we need to rethink a lot of things that are already built in.

--
Brent Royal-Gordon
Architechies

Matthew, thank you for sharing this great text!
Unfortunately was not able to read it in details yet, but after looking for proposed solution, have a quick question.

Thanks for taking a look! I’m looking forward to more feedback later.

You are proposing to have 'submodule' keyword in each file of that submodule. How can we then solve such issue: imagine that I have swift file with some code in it. I got this file from external source(repository/site), I will periodically update it with newer version for example. I want to use this file in my two submodules, but don't want to 'share' or 'make visible' types/classes/funcs etc from this file with other code outside of these submodules.

As I understand, with your suggestion the only way is have two copies of that file and append 'submodule' keyword in each. Or I understand the whole idea incorrectly? (Sorry then)

Great question. First, I want to caution that a submodule system in a programming like this is not really intended to facilitate the use of external 3rd party code. It is nothing like git submodules, for example. You’ll almost always be better off using a solution like Carthage or CocoaPods or SwiftPM.

Of course those tools do not allow you restrict the visibility of the external module within your code. It might be interesting to explore ways to restrict the import of external modules to a specific submodule scope in the future. I’m not sure how that might look but I think we could come up with something. That would allow you to integrate a third party dependency in a more robust fashion while restricting access to it in exactly the way you want.

All that aside, I’ll discuss how you could do what you asked using the tools the current proposal offers.

In order to make a file available to more than one submodule without exposing it more broadly you need to create a scope that includes both submodules that use the file as well as a submodule that contains the file. The way you create a scope like this is to place them inside a common parent (or ancestor) submodule. Here is what the structure would look like:

Parent
- Child1
- Child2
- Downloaded

Inside the file you downloaded you would need to declare `submodule Downloaded`.

Inside the parent submodule you include an export of `Downloaded` scoped to `Parent`:

scoped(Parent) export Downloaded

This export statement means that only `Parent` and its descendants are able to see `Downloaded`.

Inside the files in `Child1` and `Child2` you just say `import Downloaded` to import the symbols from the `Downloaded` submodule.

By default both `Child1` and `Child2` will be available for import everywhere in the module (modulo circular dependency and descendent restrictions).

A couple of caveats: you will not be able to see `internal` symbols declared in the file. Those will be scoped to `Downloaded` only. If you need to access `internal` symbols you would have to add a new file of your own in `Downloaded` and wrap those symbols. Second, if the file contains scope references to ancestor submodules you would not be able to use the file as-is because those ancestors would not exist in the context you placed it in your module (unless you want to be evil and couple your module structure to that of a copy-and-paste dependency).

···

On Feb 24, 2017, at 3:56 PM, Vladimir.S <svabox@gmail.com> wrote:

On 24.02.2017 22:34, Matthew Johnson via swift-evolution wrote:

I didn't expect submodules to be a part of the Swift 4 discussion. When it
came up I was pleasantly surprised. I have been thinking about the design
of a submodule system for quite a while but was planning to wait until it
was clearly in scope to draft a proposal. Now that the topic has been
introduced I decide to write down the design I've been thinking about,
along with the motivation and goals that underly it. I understand
submodules may not be in scope for Swift 4 but wanted to contribute this
design while the discussion is fresh in everyone's mind.

I am including the contents of the proposal below. You can also find it on
Github: https://github.com/anandabits/swift-evolution/blob/scope-based-submodules/proposals/NNNN-scope-based-submodules.md

I am very much looking forward to everyone's feedback!
.....

I didn't expect submodules to be a part of the Swift 4 discussion. When it came up I was pleasantly surprised. I have been thinking about the design of a submodule system for quite a while but was planning to wait until it was clearly in scope to draft a proposal. Now that the topic has been introduced I decide to write down the design I've been thinking about, along with the motivation and goals that underly it. I understand submodules may not be in scope for Swift 4 but wanted to contribute this design while the discussion is fresh in everyone's mind.

I am including the contents of the proposal below. You can also find it on Github: https://github.com/anandabits/swift-evolution/blob/scope-based-submodules/proposals/NNNN-scope-based-submodules.md

I am very much looking forward to everyone's feedback!

Matthew
Scope-based submodules

Proposal: SE-NNNN <file:///Users/Matthew/Dropbox/Matthew/Development/__notes/__swift/evolution/NNNN-scope-based-submodules.md>
Authors: Matthew Johnson <https://github.com/anandabits&gt;
Review Manager: TBD
Status: Awaiting review
Introduction

This proposal describes a submodule system based on the principle of strictly nested scopes. The design strives to follow the Swift philosophy of offering good defaults while progressively disclosing very powerful tools that can be used by experts to solve complex problems.

Motivation

Swift currently provides two kinds of entities that provide system boundaries without directly introducing a symbol: modules and files*.

Modules introduce an ABI boundary, a name boundary, and a scope boundary.
Files introduce a scope boundary that carries no additional semantics.
Swift currently lacks the ability to introduce a name and scope boundary without also introducing an ABI boundary. Such a boundary would be naturally situated halfway between in terms of strength and ceremony. The lack of such a boundary significantly limits our abiltiy to structure a large Swift program. Introducing a way to form this kind of boundary will provide a powerful tool for giving internal structure to a module.

*The important aspect of a file in Swift is the logical scope boundary it introduces. The physical file system representation is incidental to this. The appendix on file system independence discusses this in more detail.

Goals

Any discussion of submodules inevitably reveals that there are very different perspectives of what a submodule is, what problems submodules should be able to solve, etc. This section describes the design goals of this proposal in order to facilitate evaluation of both the goals themselves as well as how well the solution accomplishes those goals.

The primary goal of this proposal are to introduce a unit of encapsulation within a module that is larger than a file as a means of adding explicit structure to a large program. All other goals are subordinate to this goal and should be considered in light of it.

Some other goals of this proposal are:

Submodules should help us to manage and understand the internal dependencies of a large, complex system.
Submodules should be able to collaborate with peer submodules without necessarily being exposed to the rest of the module.
A module should not be required to expose its internal submodule structure to users when symbols are exported.
It should be possible to extract a submodule from existing code with minimal friction. The only difficulty should be breaking any circular dependencies.
Some additional non-functional requirements for the solution are:

Submodules should not negatively impact runtime performance. WMO should be able to see across submodule boundaries.
Submodules should not negatively impact build performance. Ideally they will improve build performance by giving the compiler more high-level information about internal dependencies.
Deferred goal:

It is not an immediate goal to support submodules in single-file scripts. The appendix discussing file system independence discusses some ideas that could be used to support single-file scripts in the future.
Proposed solution

There are several relatively orthogonal aspects to the design of a submodule system. A design must answer the following questions:

How is code placed in a submodule?
How are symbols in one submodule made available to another submodule or module?
How do submodules interact with access control?
This proposal diverages a little bit from the usual proposal format to faciliate discussion of alternatives within the context of each aspect of the chosen design. In each case an alternative could be substituted without compromising the overall design.

Note: This proposal uses the term “top-level submodule” to distinguish the scope of code that is not explicitly placed in a submodule from the module as a whole. The top-level submodule is in most respects identical to any other parent submodule. There are two differences: 1) it is the only submodule that can export symbols outside of the module and 2) any open or public symbols it declares are automatically exported according to their access modifier.

Placing code in a submodule

Each file is part of the top level submodule by default.

A submodule declaration may be used at the top of a file:

Only the first non-comment, non-whitespace line may contain a submodule declaration.
The submodule decalaration looks like this: submodule MySubmoduleName
A submodule declaration may not be prefixed with the module name.
Submodule names form a hierarchical path:

The fully qualified name of the submodule specified by Submodule.InnerSubmodule is: MyModuleName.Submodule.InnerSubmodule.
In this example, InnerSubmodule is a child of Submodule.
A submodule may not have the same name as any of its ancestors. This follows the rule used by types.
Submodules may not be extended. They form strictly nested scopes.

The only way to place code in a submodule is with a submodule declaration at the top of a file.
All code in a file exists in a single submodule.
A module is made up of strictly nested scoped that look like this:

<scope-based-submodules.png>
The hierarchy of nested scopes in scope-based submodules
Alternatives

Grouping mechanisms

There are several other ways to specify which submodule the top-level scope of a file is in. All of these alternatives share a crucial problem: you can’t tell what submodule your code is in by looking at the file.

The alternatives are:

Use a manifest file. This would be painful to maintain.
Use file system paths. This is too tightly coupled to physical organization. Appendix A discusses file system independence in more detail.
Leave this up to the build system. This makes it more difficult for a module to support multiple build systems.
Require all files to include a submodule declaration

We could require all files to include an explicit submodule declaration. However, that would be a breaking change and would violate the principle of progressive disclosure. Users should not need to know about submodules if they don’t use them.

Allow submodule references to explicitly state the name of the module

The module name is implicit throughout the scope of the entire module. Specifying it explicitly is redundant. Prohibiting explicit mention of the module name offers more flexibility for combining submodules into build products.

Visibility of submodules

The export statement is used to modifiy the visibiltiy of a submodule within the module. It is also used by the top-level module to publish submodules to clients of the module.

All submodules are implicitly exported with module-wide visibility by default (and hidden outside of the module by default*).
All submodules are implicitly available for export outside the module.
A submodule may use an explicit export statement to modify the visibility of a descendent submodule.

I don’t see the need for modifying the visibility of descendent modules. Can you give me an idea why its important enough to warrant all that extra syntax?

Thanks for taking a look at my proposal and offering feedback David.

You can think of descendent submodules similarly to nested types. It can be useful to have nested types that are not visible outside the scope of the parent type and the same will be true of submodules.

For example, let’s say I might have a facade submodule as the parent which delegates work to a networking submodule, a persistence submodule, etc. The module should only interact with this subsystem using the facade - it shouldn’t be interacting directly with the child submodules which are implementation details of the parent.

I want to point out that the syntax supporting submodule hiding isn’t extra syntax. It uses a subset of the syntax that the top level export feature does. Control over top-level export has been a feature of every submodule design I have seen discussed on the list.

The proposal is actually pretty conservative in terms of syntax extensions. It introduces the `submodule` declaration, the `export` statement, and parameterizes `scoped`. That’s it. There are alternatives available for each of these that don’t require any new syntax while still keeping the system as a whole intact. I obviously think the best tradeoffs are the ones I have proposed but there is room to make changes while still having a workable system if that is what the community and / or core team decides to do.

export statements are only allowed at the top level of a file.
*The exception to this is that open and public symbols in the top-level submodule are always exported exactly as declared.

Top-level export

All export statements consist of an access modifier, the export keyword, and a submodule name:

open export ChildSubmodule

When this export statement appears in the top-level submodule, ChildSubmodule becomes available for import by clients of the submodule with the fully qualified name Module.ChildSubmodule. public exports are also available. When public is used all published symbols in the exported submodule have a maximum visiblity of publicregardless of how they were declared.

Top-level public and open export statement may be modified with the following options:

A submodule may be published under a different external name using the export as NewName syntax*.
@implicit causes symbols from the submodule to be implicitly imported when the module is imported.
@inline causes the symbols from the submodule to appear as if they had been declared directly within the top-level submodule.
@implicit may be combined with renaming but @inline may not appear along with either of them.
*When a submodule is renamed for export with the as clause its internal name does not change. A submodule always has the same fully qualfied name everywhere within its module.

Here are some example export statements:

// All symbols in `Child1` are available for import by clients as `Module.Foo`
// These symbols are *not* imported automatically when a client imports the module with `import Module`
public export Child1 as Foo

// All symbols in `Child2` are available for explicit import by clients as `Module.Child2`
// The symbols are also automatically imported when a client imports the module with `import Module`
// When the symbols are imported implicitly they retain the fully qualified name prefix of `Module.Child2`
@implicit open export Child2

// All symbols in `Child3` are available for explicit import by clients as `Module.Foo.Bar`
// The symbols are also automatically imported when a client imports the module with `import Module`
// When the symbols are imported implicitly they retain the fully qualified name prefix of `Module.Foo.Bar`
@implicit open export Child3 as Foo.Bar

// All symbols in `Child4.Grandchild` appear to clients as if they had been declared
// directly in the top-level submodule.
// If the process of inlining the symbols produces duplicate symbols a compiler error is produced
// at the site of one or both of the `export` statements.
@inline public export Child4.Grandchild

// All symbols in `Child5.Grandchild` are available for explicit import by clients as `Module.Foo`
// along with the symbols declared in `Child`.
// The symbols are also automatically imported when a client imports the module with `import Module`
// When the symbols are imported implicitly they retain the fully qualified name prefix of `Module.Foo`
// As with `@inline`, when two submodules are given the same external name a duplicate symbol error may occur.
@implicit public export Child5.Grandchild as Foo
One interesting observation is that both Child1 and Child5.Grandchild are renamed to Foo. The symbols declared former is not implicitly imported by import Module but the latter is, despite having the same fully qualified name prefix.

Exports within the module

A submodule may bound the maximum visibility of any of its descendent submodules by explicitly exporting it:

// `Child1.Grandchild` may be exported by the top-level module, but only with `public` visibility.
// `Child1.Grandchild` may not be exported as `open`.
public export Child1.Grandchild

// `Child2` is exported with `internal` visibility.
// Because `internal` is scoped to the submodule level *only* the parent submodule can see `Child2`.
// No submodules except the direct parent of `Child2` (the current submodule) are allowed to `import Child2`.
// This also implies that the `Child2` may not be exported to clients because the top-level
// submodule is not able to see or reference `Child2` at all.
internal export Child2
The access modifier may specify a scope internal or greater.
Only the direct parent of a submodule may specify the internal modifier. A grandparent cannot hide a grandchild from its parent.
If a descendent includes an export statement for the same submodule, the access modifier must be no greater than the access modifier specified by the descendent. An ancestor may provide a tighter bound to visibility but may not increase visibility. An attempt to increase visibility results in an error.
Note: If a submodule is not visible none of its descendents is visible either.

Alternatives

Use import access modifiers to export a submodule

The semantics of placing a bound on the visibility of a descendent submodule is significantly different than the semantics of importing symbols from a submodule into the current lexical scope. Mixing the semantics of the two is confusing.

Restrict the visibility of a submodule to its parent unless the parent explicitly exports it.

Users should be able to use submodules without needing to place export statements in every parent submodule. Module-wide default visibility for submodules is analagous to internal default visibility for symbols.

Require all submodules to be visible module-wide.

This removes an important tool for bounded collaboration within a complex system. A parent submodule should be allowed to have a child submodule(s) which are implementation details of the parent and not exposed to the rest of the module.

IMHO, the implementation details of a submodule should live in that submodule, not in a child submodule.

Sometimes yes. But there are different granularities of implementation details. The ability to hide child submodules gives us the ability to hide a coarser-grained implementation detail which has its own internals that should not be visible to the parent it is performing work on behalf of.

Shouldn’t the main purpose of submodules be to help organise publicly available code into name-spaces, and not add to add scoping mechanisms for implementation?

Not at all. Modules and package managers are for organizing publicly available code. Submodules are about structuring the code within our own module.

For example, looking at C#’s System.Xml library, the namespaces clearly help to separate code, like serialisation classes from more mundane Xml reading classes, etc...

System.Xml
System.Xml.Linq
System.Xml.Resolvers
System.Xml.Schema
System.Xml.Serialization
System.Xml.Serialization.Advanced
System.Xml.Serialization.Configuration
System.Xml.XmlConfiguration
System.Xml.XPath
System.Xml.Xsl
System.Xml.Xsl.Runtime

You can do this with my proposal. You can have arbitrarily nested submodules with hierarchical names that look exactly like that.

Allow renaming to be used by export statements within the module.

A submodule should have the same fully qualified name everywhere it is used within a single module, whether that be the declaring module or a client module. The declaring module and client modules may see different names, but each sees a name that is consistent fully qualified name everywhere the submodule is referenced.

Allow @inline to be used by export statements within the module.

As with renaming, a symbol should have a single fully qualified name everywhere within a single module.

Allow @implicit to be used by export statements within the module.

This would reduce the visibility of internal dependencies. If we find that import-per-submodule becomes boilerplate-y this is an easy feature to add later.

Importing submodules

Submodules are imported in exactly the same way as an external module by using an import statement. There are a few additional details that are not applicable for external modules:

Circular imports are not allowed.
A submodule may not import any of its ancestors.
Relative child and sibling names are allowed using the same rules that apply to nested types.
Access control

An access modifier applies an upper bound to the scope in which a symbol or submodule is visible. With the introduction of submodules, internal now applies at the level of a submodule: only the current submodule may see an internal entity. We need a new way to specify module-wide visibility.

This proposal builds on Option 2 in the proposal Fix Private Access Levels which reverts private to the Swift 2 meaning (equivalent to fileprivate) and uses scoped for the Swift 3 scoped access feature. It does this by allowing the scoped access modifier to be parameterized with a scope reference. By defaults it references the scope in which it appears, but any ancestor scope may be specified as a parameter.

The paremeterization of the scoped access modifier provides a simple yet powerful way for a submodule to bound the visibility of a descendent.

Some examples of using scoped exports are:

submodule Parent

// `Grandparent` and all of its descendents can see `Child1` (fully qualified: `Grandparent.Parent.Child1`)
// This reads: `Child1` is scoped to `Grandparent`.
scoped(Grandparent) export Child1

// `Child2` is visible throughout the module but may not be exported for use by clients.
// This reads: `Child2` is scoped to the module.
scoped(module) export Child2
With parameterization, scoped has the power to specify all access levels that Swift has today:

`scoped` == `private` (Swift 3)
`scoped(file)` == `private` (Swift 2 & 4?) == `fileprivate` (Swift 3)
`scoped(submodule)` == `internal`
`scoped(public) scoped(internal, inherit)`* == `public`
`scoped(public)` == `open`
This design is a direct generalization of the principle underlying Swift’s existing access control system. It unifies the semantics of the system under the single elegant mechanism of ancestor scope references.

While it is possible to specify all access levels using scoped that is not recommended. The aliases public, private(Swift 2) and internal provide excellent default access levels that don’t require a user to think about scope hierarchies. Using the default access levels when possible calls extra attention to cases where a different choice was made.

*This is a conceptual model. This proposal does not introduce the inherit or override parameter to access modifiers. It could be added in the future as a way to bound inheritance within a module. It would work similarly to private(set) does in Swift today.

Like we’ve had time to discuss, I’m really not sold on this solution as it adds a fair amount of complexity, and complexity is exactly what people have been complaining about in the current model.

It introduces complexity in one sense, but reduces it in another by providing a consistent semantic for access modifiers. I really think the biggest mistake was that `private` was a bad “soft default”. This caused lots of confusion around when to use `private` and when to use `fileprivate`. If we had left `private` alone and introduced `scoped` last year I don’t think we would have seen nearly so much grumbling. People who didn’t want to use it just wouldn’t. But instead, they were “forced” to use `fileprivate` and didn’t like that.

The complexity I’m not sure we need:

new keyword

No, we have the same number of keywords because `fileprivate` goes away.

more than 1 way to express the same semantics

We have syntactic sugar and shorthand for many things in the language. I don’t think this is that different. I would not object to banning the `scoped` spellings that have a shorthand equivalent if people felt strongly about that. They could still be taught in the context of scoped access.

the parametrisation of the keyword

We already have parameterized keywords, including parameterized access modifiers - `private(set)` is an example. Parameterization of a keyword is not any more conceptually complex than a function call, which are pervasive in the language.

The most important thing about the design I have proposed is that it stays out of your way until you decide you need to use the tools available. Every aspect is progressively disclosed when it becomes necessary to solve a problem and not before.

That said, I definitely don’t want to see people get hung up on the access control section of this proposal. The big picture works fine with a more limited access control system, it just isn’t as flexible.

Let’s start with the goals section. Do you think that’s in the right ballpark? If so, then we should discuss the alternatives for each aspect of the proposal. If not we should focus on where we have different opinions about the goals.

Aside

The parameterization of scoped also allows us to reference other scopes that we cannot in today’s system, specifically extensions: scoped(extension) and outer types: scoped(TypeName).

Alternatives

If we don’t adopt the approach of parameterizing scoped our options for access control include:

Submodules are only allowed to see public and open symbols from other submodules

A module-wide scope is highly desirable. People might avoid using submodules if this is not available.

This approach also creates a lot more friction when refactoring. A possible workaround to the lack of a module-wide scope in this system is to place code in a non-exported submodule and declare symbols public. Even with the workaround, extracting a submodule may not always be possible or desirable and the public access modifiers required would be misleading. It would be much better to be able to state our intent directly.

Use internal to cover the whole module and private to cover a submodule

One suggestion that has appeared is the idea of removing fileprivate and making private be submodule-wide. internal would remain module-wide. This is too coarse - many people want a file-level scope.

I agree that this is a bad idea.

internal is Swift’s default access modifier. A symbol with default access modifier should not be able to cross a submodule boundary implicitly.

Add the moduleinternal access modifier

This is about as ugly as fileprivate.

What about the alternative of not having a submodule scope at all? Submodules would see all internal access in all submodules, as long as it’s imported at the top of the file.

That’s certainly an option. It’s certainly not one I would choose, but as I said earlier, this proposal will work with whatever access modifiers we want. Changing that part of the proposal only changes what you have the ability to express.

I think the divide comes from the fact that I ant “namespaces” as a name grouping mechanism and you want “submodules” as a scope grouping mechanism.

Yes, these are at odds with each other and clearly there are two camps. I elaborate on why I don’t want namespaces in the second appendix.

···

On Feb 24, 2017, at 4:57 PM, David Hart <david@hartbit.com> wrote:

On 24 Feb 2017, at 20:34, Matthew Johnson via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:
Detailed design

Export errors

Multiple exports of the same submodule

If a submodule exports the same descendent more than once and the semantics of the declarations are not identical an error is produced.

Symbol flattening

When a submodule is exported by the top-level module using the @inline attribute it is possible that there will be conflicting symbol definitions in the child and the top-level submodule (or other inlined submodules). This results in a compiler error at the site of the conflicing @inline export statements.

Overlapping renames

As with flattening, when two or more submodules are given the same external name symbol conflicts are possible. This also results in a compiler error at the site of the conflicting export as statements.

Access errors during export if the specified access modifier exceeds maximum

An error is produced when an export statement includes an access modifier greater than the bound provided for the exported submodule by a descendent of the exporting submodule.

Source compatibility

This proposal is purely additive. That said, it would be a breaking change for the standard library to move existing API into an externally visible submodule.

Effect on ABI stability

This proposal is purely additive. That said, it would be a breaking change for the standard library to move existing API into an externally visible submodule.

Effect on API resilience

This proposal is purely additive.

Future directions

Selective export and import

The ability to import and export individual symbols would be a very nice to have.

Scoped import

The ability to import modules, submodules, and symbols into any lexical scope would be nice to have.

Bounded inheritance

It could be useful to have the ability to bound inheritance within a module. This could be accomplished by inroducing inherit and override parameters for access modifiers (which would work similarly to the existing set parameter).

Appendix A: file system independence

The submodule design specified by this proposal is file system independent. The only relationship it has with the physical file system is that a file introduces an anonymous scope boundary which is referenced by scoped(file) or fileprivate (Swift 3) or private (Swift 2 and 4?).

The logical role of a “file” in this design is to provide a boundary of encapsulation that is even lighter weight than a submodule: it doesn’t hide names. All declarations are implicitly available not only within the file but also across the file boundary (modulo access control). Files are to submodules as submodules are to modules.

If a future version of Swift were to eliminate files in favor of some kind of code browser it would still be very useful to have the ability to form a pure scope boundary (with no additional semantics). A scope declaration could be used to do this. Scope declarations could have an optional (or required) name and could even be nested. In this system privatewould reference the nearest anonymous ancestor scope declaration (or would be removed if we don’t allow anonymous scopes).

The logical structure of this design can be directly translated into a grammar that could be represented directly with syntax. Such a grammer could be used to support scripts with submodules. An example follows:

// A module contans a single implicit, anonymous submodule.
// submodule {
  // A submodule may contain `scope` declarations (i.e. files) as well as other submodules.
  // An anonymous scope is equivalent to a file in current Swift.
  // If we introduce lexical scopes we would probably require them to be named explicitly.
  // This example uses the anonymous scope in order to most closely match the role files play in the current system.
  // Because `scope` does not provide a name boundary all names declared in one scope
  // are visible in other scopes (modulo access control)
  scope {
      // Top-level declarations go here.
      // This is equivalent to the top level of a file in Swift today.
      // It is also equivalent to the top level of a file that does not contain
      // a `submodule` declaration in this proposal.
      
      // It would be possible to allow nested, named scopes.
      // A scope name participates in the scope and name hierachies.
      // However, it does not form a name boundary like a submodule does.
      scope Named {
        // This declares the static variable `Named.foo`
        // `scoped(file)` references the nearest anonymous ancestor scope.
        // It is used in this example for specificity.
        // Real code would use the alias `private` or `fileprivate`
        // If we introduce explicit scope syntax we would probably want a better name to refer
        // to the nearest anonymous scope than `file` or we may just require all scopes to have a name.
        scoped(file) var foo: String
      }
      // `Named.foo` is visible here
  }
  // `Named.foo` is not visible here.
  
  submodule Foo {}
  submodule Baz {}
  submodule Buzz {
    // Equivalient to a file in current Swift.
    scope {
      // submodule declarations go here.
      // This is equivalent to the top level scope of a file that contains the `submodule Foo` declaration.
    }
    scope {}
    submodule Baz {}
  }
//}
Appendix B: namespace style submodules

It is possible to design a system that allows a name boundary to be formed without also forming a scope boundary. A natural consequence of this is that symbols may be placed into a namespace-style submodule in many (unlimited) scopes via extension (even extension outside the module is theoretically possible). Allowing this runs contray to both of the two primary goals of this proposal (encapsulation and structure).

Allowing a submodule to be extended in multiple scopes precludes the possibility of submodule internal visibility. A submodule internal access modifier could still be defined but it would not provide the guarantee it purports to. The submodule can be opened by extension anywhere within the module. If a lazy developer wants to access a submodule internal symbol from a distant subsytem all they need to do is add an extension and wrap the submodule internal symbol with a new symbol offering higher visibility*. In such a system there is the same wide gap between file scope and module scope that exists today.

Allowing a submodule to be extended in multiple scopes precludes the ability to introduce real structure to a module. We are able to introduce structure to the names but not the module itself. The structure of a submodule in such a system may be widely dispersed throughout the module. It is not part of a strictly hierarchical structure of scopes which each having a single designated location within the larger structure.

What you do get from name boundaries that do not also form a scope boundary is a soft form of symbol hiding (soft because all submodules are available for import or extension anywhere within the program). This does provide some value, but not nearly as much value as is provided by a name boundary that is accompanied by a scope boundary.

Another downside to namespace-style submodules that are open to extension is that they are much less likely to facilitate improved build performance because they don’t add any physical structure to the system.

Finally, lexical submodules have the potential to be confusing. If submodules form a name boundary (even a soft one) an import statement is required to access the symbols declared inside a submodule. Is code that surrounds a lexical submodule declaration able to see the symbols it declares without importing them? Most developers will expect the symbols to be available. It is probably necessary to make an exception to name boundary for the surrounding lexical context. However, if an exception is made then this system relies heavily on a file to provide a bound to the implicit symbol import.

*It is worth observing that the ability to violate encapsulation via extension (or subclassing) is one of the primary reasons Swift does not offer type-based access modifiers such as typeprivate or protected. The do not offer true encapsulation at all. They are a statement of intent that cannot really be verified in the way that is desired. They form a permeable rather than a hard boundary.

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

Wonderful comments. I really enjoy your take on submodules which keeps most of the power while keeping the simplicity. Comments below:

Scope-based submodules

   • Proposal: SE-NNNN
   • Authors: Matthew Johnson
   • Review Manager: TBD
   • Status: Awaiting review

Well, this is certainly comprehensive! Sorry about the delay in answering; I've been hosting a house guest and haven't had a lot of free time.

The primary goal of this proposal are to introduce a unit of encapsulation within a module that is larger than a file as a means of adding explicit structure to a large program. All other goals are subordinate to this goal and should be considered in light of it.

I agree with this as the primary goal of a submodule system.

Some other goals of this proposal are:

   • Submodules should help us to manage and understand the internal dependencies of a large, complex system.
   • Submodules should be able to collaborate with peer submodules without necessarily being exposed to the rest of the module.
   • A module should not be required to expose its internal submodule structure to users when symbols are exported.
   • It should be possible to extract a submodule from existing code with minimal friction. The only difficulty should be breaking any circular dependencies.

One goal I don't see mentioned here is "segment the API surface exposed to importing code". The `UIGestureRecognizerSubclass` use case has been thoroughly discussed, but I think there are probably a lot of cases where there are two "sides" to an API and it'd often be helpful to hide one unless it's needed. `URLProtocol` and `URLProtocolClient` come to mind; the many weird little classes and symbols related to `NSAtomicStore` and `NSIncrementalStore` might be another.

(I'm not necessarily suggesting that the Foundation and Core Data overlays should move these into submodules—I'm suggesting that, if they were implemented in a Swift with a submodule feature, they would be candidates for submodule encapsulation.)

Submodule names form a hierarchical path:

   • The fully qualified name of the submodule specified by Submodule.InnerSubmodule is: MyModuleName.Submodule.InnerSubmodule.
   • In this example, InnerSubmodule is a child of Submodule.
   • A submodule may not have the same name as any of its ancestors. This follows the rule used by types.

Does being in a nested submodule have any semantic effect, or is it just a naming trick?

Submodules may not be extended. They form strictly nested scopes.

   • The only way to place code in a submodule is with a submodule declaration at the top of a file.
   • All code in a file exists in a single submodule.

I'm a big supporter of the 1-to-N submodule-to-file approach.

Agreed.

There are several other ways to specify which submodule the top-level scope of a file is in. All of these alternatives share a crucial problem: you can’t tell what submodule your code is in by looking at the file.

The alternatives are:

   • Use a manifest file. This would be painful to maintain.
   • Use file system paths. This is too tightly coupled to physical organization. Appendix A discusses file system independence in more detail.
   • Leave this up to the build system. This makes it more difficult for a module to support multiple build systems.

I'm going to push back on this a little. I don't like the top-of-file `submodule` declaration for several reasons:

   1. Declarations in a Swift file are almost always order-independent. There certainly aren't any that must be the first thing in the file to be valid.

   2. Swift usually keeps configuration stuff out of source files so you can copy and paste snippets of code or whole files around with minimum fuss. Putting `submodule` declarations in files means that developers would need to open and modify those files if they wanted to copy them to a different project. (It's worth noting that your own goal of making it easy to extract submodules into separate modules is undermined by submodule declarations inside files.)

   3. However you're organizing your source code—whether in the file system, an IDE project, or whatever else—it's very likely that you will end up organizing files by submodule. That means either information about submodules will have to be specified twice—once in a canonical declaration and again in source file organization—and kept in sync, or IDEs and tooling will have to interpret the `submodule` declarations in source files and reflect that information in their UIs.

   4. Your cited reason for rejecting build system-based approaches is that "This makes it more difficult for a module to support multiple build systems", but Swift has this same problem in *many* other parts of its design. For instance, module names and dependencies are build system concerns, despite the fact that this makes it harder to support multiple build systems. I can only conclude that supporting multiple build systems with a single code base is, in the long term, a non-goal, presumably by improving the Xcode/SwiftPM story in some way.

I'm still a fan of build-system-based approaches because I think they're better about these issues. The only way that they're worse is that—as you note—it may not be clear which submodule a particular file is in. But I think this is basically a UI problem for editors.

I'm also a big fan of the build-system or file-system approach, for exactly the same reasons. The copy/pasting argument is important IMHO. For example, the suggestion which has been discussed a lot on here to make private change to submodule scope when inside a submodule would have the same negative effect on the ability to copy paste code between a submodule and top-module.

Top-level export

All export statements consist of an access modifier, the export keyword, and a submodule name:

open export ChildSubmodule

Is the access control keyword mandatory?

If we do indeed use `submodule` statements, could we attach the attributes to them, rather than having a separate statement in a different file?

What does it mean if a `public` or `open` symbol is in a submodule which is not `export`ed?

   • A submodule may be published under a different external name using the export as NewName syntax*.

What's the use case for this feature?

   • @implicit causes symbols from the submodule to be implicitly imported when the module is imported.
   • @inline causes the symbols from the submodule to appear as if they had been declared directly within the top-level submodule.

So if you write `@implicit public export Bar` in module `Foo`, then writing `import Foo` also imports `Foo.Bar.Baz` *as* `Foo.Bar.Baz`, whereas `@inline public export Bar` copies `Foo.Bar.Baz` into `Foo`, so it imports as `Foo.Baz`?

What's the use case for supporting both of these behaviors?

Exports within the module

A submodule may bound the maximum visibility of any of its descendent submodules by explicitly exporting it:

I'm not sure how valuable this feature is in this kind of submodule design.

***

To avoid being coy, here's the export control model that *I* think would make the most sense for this general class of submodule system designs:

1. A submodule with `public` or `open` symbols is importable from outside the module. There is no need to separately mark the submodule as importable.

I had not thought of that but it's very elegant.

2. Normally, references within the module to submodule symbols need to be prefixed with the submodule name. (That is, in top-level `Foo` code, you need to write `Bar.Baz` to access `Foo.Bar.Baz`). As a convenience, you can import a submodule, which makes the submodule's symbols available to that file as though they were top-level module symbols.

3. When you import a submodule, you can mark it with `@exported`; this indicates that the symbols in that submodule should be aliased and, if `public` or `open`, re-exported to other modules.

Could you explain this in more detail?

4. There are no special facilities for renaming submodules or implicitly importing submodules.

Importing submodules

Submodules are imported in exactly the same way as an external module by using an import statement.

Okay, but what exactly does importing *do*? Set up un-prefixed private aliases for the submodule's internal-and-up APIs?

There are a few additional details that are not applicable for external modules:

   • Circular imports are not allowed.

Why not? In this design, all submodules are evaluated at once, so I'm not sure why circular imports would be a problem.

// `Grandparent` and all of its descendents can see `Child1` (fully qualified: `Grandparent.Parent.Child1`)
// This reads: `Child1` is scoped to `Grandparent`.

scoped(Grandparent) export Child1

// `Child2` is visible throughout the module but may not be exported for use by clients.
// This reads: `Child2` is scoped to the module.

scoped(module) export Child2

With parameterization, scoped has the power to specify all access levels that Swift has today:

`scoped` == `private` (Swift 3)
`scoped(file)` == `private` (Swift 2 & 4?) == `fileprivate` (Swift 3)
`scoped(submodule)` == `internal`
`scoped(public) scoped(internal, inherit)`* == `public`
`scoped(public)` == `open`

The parameterization of scoped also allows us to reference other scopes that we cannot in today’s system, specifically extensions: scoped(extension) and outer types: scoped(TypeName).

What is the purpose of creating more verbose aliases for existing access levels? I can't think of one, which means that these are redundant.

And if we remove them as redundant, the remaining access control levels look like:

   scoped
   private
   scoped(TypeName)
   internal
   scoped(SomeModule)
   scoped(module)
   scoped(extension)
   public
   open

There's just no logic to the use of the `scoped` keyword here—it doesn't really mean anything other than "we didn't want to assign a keyword to this access level".

***

I think we need to go back to first principles here. The reason to introduce a new access level is that we believe that a submodule is a large enough unit of code that it will simultaneously need to encapsulate some of its implementation details from other submodules, *and* have some of its own implementation details encapsulated from the rest of the submodule.

That's where I disagree. Why would Swift need that level of differentiation and not C# or Java, which have had packages and namespaces without it.

Thus, we need at least three access levels within a submodule: one that exposes an API to other submodules, one that exposes an API throughout a submodule, and one that exposes it to only part of a submodule.

What we do *not* need is a way to allow access only from certain other named submodules. The goal is to separate external and internal interfaces, not to micromanage who can access what.

Basically, that means we need one of two things. Keeping all existing keywords the same—i.e., not removing either `private` or `fileprivate`— and using `semi-` as a placeholder, we want to either have:

   private: surrounding scope
   fileprivate: surrounding file
   semi-internal: surrounding submodule
   internal: surrounding module
   public: all modules (no subclassing)
   open: all modules (with subclassing)

Or:

   private: surrounding scope
   fileprivate: surrounding file
   internal: surrounding submodule
   semi-public: surrounding module
   public: all modules (no subclassing)
   open: all modules (with subclassing)

The difference between the two is that, with `semi-internal` below `internal`, submodule APIs are exposed by default to other submodules; with `semi-public` above `internal`, submodule APIs are encapsulated by default from other submodules.

I think encapsulating by default is the right decision, so we want the `semi-public` design. But there's also a second reason to use that design: We can anticipate another use case for it. The library resilience design document discusses the idea of "resilience domains"—groups of libraries whose versions are always matched, and which therefore don't need to use resilient representations of each others' data structures—and the idea of having "SPIs", basically APIs that are only public to certain clients. I think these ideas could be conflated, so that a semi-public API would be available both to other submodules in the module and to other libraries in your resilience domain, and that this feature could be used to expose SPIs.

So, that leaves an important question: what the hell do you call this thing? My best suggestions are `confidential` and `privileged`; in the context of information, these are both used to describe information which *is* shared, but only within a select group. (Think, for instance, of attorney-client privilege: You can share this information with your lawyer, but not with anyone else.)

So in short, I suggest adding a single access level to the existing system:

   private
   fileprivate
   internal
   confidential/privileged
   public
   open

This is orthogonal to any other simplification of the access control system, like removing `private` or `fileprivate`.

Yes, I would still like to see scope private go away.

···

Sent from my iPhone
On 1 Mar 2017, at 07:55, Brent Royal-Gordon via swift-evolution <swift-evolution@swift.org> wrote:

On Feb 24, 2017, at 11:34 AM, Matthew Johnson via swift-evolution <swift-evolution@swift.org> wrote:

Appendix A: file system independence

I think we need to decide: Is a translation unit of some sort—whether it's a physical on-disk file or some simulacrum like a database record or just a separate string—something intrinsic to Swift? I think it should be; it simplifies a lot of parts of the language that would otherwise require nesting and explicit scoping.

If translation units are an implicit part of Swift, then this section is not really necessary. If translation units aren't, then we need to rethink a lot of things that are already built in.

--
Brent Royal-Gordon
Architechies

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Scope-based submodules

  • Proposal: SE-NNNN
  • Authors: Matthew Johnson
  • Review Manager: TBD
  • Status: Awaiting review

Well, this is certainly comprehensive! Sorry about the delay in answering; I've been hosting a house guest and haven't had a lot of free time.

Hi Brent, thanks for providing feedback on the proposal!

The primary goal of this proposal are to introduce a unit of encapsulation within a module that is larger than a file as a means of adding explicit structure to a large program. All other goals are subordinate to this goal and should be considered in light of it.

I agree with this as the primary goal of a submodule system.

Some other goals of this proposal are:

  • Submodules should help us to manage and understand the internal dependencies of a large, complex system.
  • Submodules should be able to collaborate with peer submodules without necessarily being exposed to the rest of the module.
  • A module should not be required to expose its internal submodule structure to users when symbols are exported.
  • It should be possible to extract a submodule from existing code with minimal friction. The only difficulty should be breaking any circular dependencies.

One goal I don't see mentioned here is "segment the API surface exposed to importing code”.

Yeah, this is certainly a goal. I didn’t explicitly state it because it feels intrinsic to any submodule solution. Do you think it’s important to call it out explicitly?

The `UIGestureRecognizerSubclass` use case has been thoroughly discussed, but I think there are probably a lot of cases where there are two "sides" to an API and it'd often be helpful to hide one unless it's needed. `URLProtocol` and `URLProtocolClient` come to mind; the many weird little classes and symbols related to `NSAtomicStore` and `NSIncrementalStore` might be another.

Submodules could certainly address these use cases, but I’m not sure they are the best solution we can come up with. I’ve been kicking around an idea I’m calling “symbol groups”. A symbol group would be a super lightweight way to isolate symbols that should not be imported by default. When you do need them you would say something like this:

`import UIKit including UIGestureRecognizerSubclass`

The basic idea is that these symbols should be explicitly requested on import, but otherwise are considered part of the same submodule. There are two reasons I think a really lightweight mechanism like this is a better solution for this use case. First, a submodule feels like a pretty heavy solution to isolate a handful of symbols that are all related to the same type. Second, a submodule establishes a scope boundary and that will often be undesirable and complicate the implementation. Symbol groups avoid both of those problems.

I haven’t given too much thought to how symbol groups would be declared and how symbols would be placed in them. What do you think of the concept?

(I'm not necessarily suggesting that the Foundation and Core Data overlays should move these into submodules—I'm suggesting that, if they were implemented in a Swift with a submodule feature, they would be candidates for submodule encapsulation.)

Submodule names form a hierarchical path:

  • The fully qualified name of the submodule specified by Submodule.InnerSubmodule is: MyModuleName.Submodule.InnerSubmodule.
  • In this example, InnerSubmodule is a child of Submodule.
  • A submodule may not have the same name as any of its ancestors. This follows the rule used by types.

Does being in a nested submodule have any semantic effect, or is it just a naming trick?

It does have a semantic effect. Ancestor submodules are allowed to bound the visibility of descendants. They also form a scope boundary that I would like to be able to reference when bounding the visibility of a submodule as well as in access modifiers.

Submodules may not be extended. They form strictly nested scopes.

  • The only way to place code in a submodule is with a submodule declaration at the top of a file.
  • All code in a file exists in a single submodule.

I'm a big supporter of the 1-to-N submodule-to-file approach.

There are several other ways to specify which submodule the top-level scope of a file is in. All of these alternatives share a crucial problem: you can’t tell what submodule your code is in by looking at the file.

The alternatives are:

  • Use a manifest file. This would be painful to maintain.
  • Use file system paths. This is too tightly coupled to physical organization. Appendix A discusses file system independence in more detail.
  • Leave this up to the build system. This makes it more difficult for a module to support multiple build systems.

I'm going to push back on this a little. I don't like the top-of-file `submodule` declaration for several reasons:

  1. Declarations in a Swift file are almost always order-independent. There certainly aren't any that must be the first thing in the file to be valid.

Sure, but we also don’t put any semantic information outside of source files today either.

  2. Swift usually keeps configuration stuff out of source files so you can copy and paste snippets of code or whole files around with minimum fuss. Putting `submodule` declarations in files means that developers would need to open and modify those files if they wanted to copy them to a different project. (It's worth noting that your own goal of making it easy to extract submodules into separate modules is undermined by submodule declarations inside files.)

The goal is to extract submodules from code within the same module. The files are going to need to be configured as part of a new submodule one way or another. I don’t see the declaration as burdensome.

  3. However you're organizing your source code—whether in the file system, an IDE project, or whatever else—it's very likely that you will end up organizing files by submodule. That means either information about submodules will have to be specified twice—once in a canonical declaration and again in source file organization—and kept in sync, or IDEs and tooling will have to interpret the `submodule` declarations in source files and reflect that information in their UIs.

The problem with using the file system is that even if I physically organize my files by submodule I might want to have a submodule with sub-folders all containing code that is in that submodule (not in descendent submodules). This makes it tricky to use a file system convention for defining submodule boundaries.

If you put the information in the build system it is distant from the code. I would prefer to just tell the build system where to find the source files for my module and have it work out how they are organized into submodules. Requiring the declaration to be at the top of the file should help with this.

  4. Your cited reason for rejecting build system-based approaches is that "This makes it more difficult for a module to support multiple build systems", but Swift has this same problem in *many* other parts of its design. For instance, module names and dependencies are build system concerns, despite the fact that this makes it harder to support multiple build systems. I can only conclude that supporting multiple build systems with a single code base is, in the long term, a non-goal, presumably by improving the Xcode/SwiftPM story in some way.

Fair enough. This was a secondary point in my mind. The primary point is that I think semantically relevant information belongs in source files. In this case, it belongs at the *top* of the source file so a reader has the necessary context before seeing the rest of the code.

I'm still a fan of build-system-based approaches because I think they're better about these issues. The only way that they're worse is that—as you note—it may not be clear which submodule a particular file is in. But I think this is basically a UI problem for editors.

Comments above aside, I would not want to see the mechanism for placing files in a submodule sink the proposal. I am happy to defer to the judgment of the core team and the community on this issue.

Top-level export

All export statements consist of an access modifier, the export keyword, and a submodule name:

open export ChildSubmodule

Is the access control keyword mandatory?

It is mandatory. It specifies the bound of the export.

If we do indeed use `submodule` statements, could we attach the attributes to them, rather than having a separate statement in a different file?

I considered this. The problem is that the `submodule` statement is specified in every file that resides in the submodule. We don’t want to require annotations to be repeated everywhere.

Secondarily, the proper way to think about this is that the submodule is a declaration introduced in the scope of its parent, just like any declaration. However, in this case the scopes are logical rather than physical / lexical. This is why the parent specifies annotations using an `export` statement. The `export` statement in the parent is semantically equivalent to a declaration. We allow the declaration to be elided if the parent doesn’t have any annotations to add because the children already state the name of the submodule.

What does it mean if a `public` or `open` symbol is in a submodule which is not `export`ed?

It means exactly the same thing as having a `public` or `open` symbol in a `private` type. The availability is bounded by the containing scope.

  • A submodule may be published under a different external name using the export as NewName syntax*.

What's the use case for this feature?

One of the stated goals of the proposal is "A module should not be *required* to expose its internal submodule structure to users when symbols are exported.”

The idea is that submodules form scope boundaries inside the module. The structure of these scope boundaries is really an implementation detail. Outside the module they form an interface provided to users of the module. The interface exposed to users should not have to be coupled to implementation details. Renaming allows us to freely use submodules to form scope boundaries in the implementation while still exposing the preferred interface externally to users.

This also allows the submodule structure of a module to evolve over time without breaking users. The external interface can be stable while the internal structure changes to meet the needs of the implementation.

  • @implicit causes symbols from the submodule to be implicitly imported when the module is imported.
  • @inline causes the symbols from the submodule to appear as if they had been declared directly within the top-level submodule.

So if you write `@implicit public export Bar` in module `Foo`, then writing `import Foo` also imports `Foo.Bar.Baz` *as* `Foo.Bar.Baz`, whereas `@inline public export Bar` copies `Foo.Bar.Baz` into `Foo`, so it imports as `Foo.Baz`?

What's the use case for supporting both of these behaviors?

That’s the idea. It seems like an arbitrary and unnecessary limitation to couple implicit import to also inlining the symbols. I would be willing to let this go if there was enough resistance that it compromised the reception of the proposal as a whole. It could always be added later.

Exports within the module

A submodule may bound the maximum visibility of any of its descendent submodules by explicitly exporting it:

I'm not sure how valuable this feature is in this kind of submodule design.

I think this is a very important feature. I would like to be able to have peer submodules collaborate in ways that are not visible beyond the parent.

One of the big goals I have is to allow us to realize the logical benefits of encapsulation modules provide without the physical separation that results in additional build complexity and performance degradation (at least until we have whole program optimization and maybe event then). Some modules can be quite large. The ability to use layers of encapsulation to structure a large module is invaluable IMO. Most of us do this logically already. We just don’t have a way to state these boundaries in the language.

***

To avoid being coy, here's the export control model that *I* think would make the most sense for this general class of submodule system designs:

1. A submodule with `public` or `open` symbols is importable from outside the module. There is no need to separately mark the submodule as importable.

This is an interesting approach. But it runs against the grain of layered scopes of encapsulation.

I also *like* the idea of having an `Exports.swift` file or something like that at the top level of a module which lists all of the submodules visible externally and performs any name mapping required. This will require minimal maintenance and provide important information to newcomers on a project.

2. Normally, references within the module to submodule symbols need to be prefixed with the submodule name. (That is, in top-level `Foo` code, you need to write `Bar.Baz` to access `Foo.Bar.Baz`). As a convenience, you can import a submodule, which makes the submodule's symbols available to that file as though they were top-level module symbols.

One of the goals I have for this proposal is to help identify internal dependencies between submodules. I think requiring `import` statements helps to facilitate this goal. But maybe I could be convinced that requiring fully qualified names is enough. I’m not sure though.

3. When you import a submodule, you can mark it with `@exported`; this indicates that the symbols in that submodule should be aliased and, if `public` or `open`, re-exported to other modules.

I don’t understand what the purpose of this would be given your point 1 above. I definitely don’t like the idea of a symbol being exposed outside the module with more than one fully qualified name.

4. There are no special facilities for renaming submodules or implicitly importing submodules.

Is there a reason you think renaming is not important? Do you really believe that the names users see should be coupled to the scope boundaries a module uses in its implementation?

Importing submodules

Submodules are imported in exactly the same way as an external module by using an import statement.

Okay, but what exactly does importing *do*? Set up un-prefixed private aliases for the submodule's internal-and-up APIs?

This and also make the APIs available. The symbols in other submodules are not available without import.

There are a few additional details that are not applicable for external modules:

  • Circular imports are not allowed.

Why not? In this design, all submodules are evaluated at once, so I'm not sure why circular imports would be a problem.

One of the goals of submodules is to help manage internal dependencies. Is there a specific reason you believe circular imports should be allowed? Do you have use cases that are not possible without circular dependencies?

// `Grandparent` and all of its descendents can see `Child1` (fully qualified: `Grandparent.Parent.Child1`)
// This reads: `Child1` is scoped to `Grandparent`.

scoped(Grandparent) export Child1

// `Child2` is visible throughout the module but may not be exported for use by clients.
// This reads: `Child2` is scoped to the module.

scoped(module) export Child2

With parameterization, scoped has the power to specify all access levels that Swift has today:

`scoped` == `private` (Swift 3)
`scoped(file)` == `private` (Swift 2 & 4?) == `fileprivate` (Swift 3)
`scoped(submodule)` == `internal`
`scoped(public) scoped(internal, inherit)`* == `public`
`scoped(public)` == `open`

The parameterization of scoped also allows us to reference other scopes that we cannot in today’s system, specifically extensions: scoped(extension) and outer types: scoped(TypeName).

What is the purpose of creating more verbose aliases for existing access levels? I can't think of one, which means that these are redundant.

I’m going to request that we move this topic to the thread I started for my new access control proposal. It elaborates extensively on the rationale. We can discuss further there.

As far as this proposal goes, I really don’t want access control to get in the way of the larger proposal. The basic design works regardless of what access control looks like. It would be a shame if we don’t have both submodule and module wide scopes available but that is somewhat orthogonal to the larger design of submodules.

The aspect of the proposal that relies most heavily on this is exports bounded by an ancestor submodule. As you have already noted, the basic design still works without that feature. It would also be possible to redesign that feature so that it doesn’t rely on access control. I think that would be unfortunate but it is possible.

And if we remove them as redundant, the remaining access control levels look like:

  scoped
  private
  scoped(TypeName)
  internal
  scoped(SomeModule)
  scoped(module)
  scoped(extension)
  public
  open

There's just no logic to the use of the `scoped` keyword here—it doesn't really mean anything other than "we didn't want to assign a keyword to this access level”.

I’m looking forward to your feedback on the new access control proposal.

***

I think we need to go back to first principles here. The reason to introduce a new access level is that we believe that a submodule is a large enough unit of code that it will simultaneously need to encapsulate some of its implementation details from other submodules, *and* have some of its own implementation details encapsulated from the rest of the submodule. Thus, we need at least three access levels within a submodule: one that exposes an API to other submodules, one that exposes an API throughout a submodule, and one that exposes it to only part of a submodule.

What we do *not* need is a way to allow access only from certain other named submodules. The goal is to separate external and internal interfaces, not to micromanage who can access what.

I think the difference in our perspectives is that you’re viewing submodules as a basically flat system which maybe have names that form a hierarchy. I’m viewing submodules as a true hierarchy of scopes, kind of like russian dolls. This is very different from something like `friend` in C++. You are *not* allowed to give access to any arbitrary submodule. You are not even allowed to give access to any *single* submodule. What you are allowed to do is bound visibility to a *family* of submodules that all have a common ancestor (the bound).

This is not micromanaging who gets to access what. It is saying that we can form scopes that increase in size from files to submodules to parent submodules to the entire module. I want truly hierarchical submodule scopes, not just a flat sea of submodule scopes.

Basically, that means we need one of two things. Keeping all existing keywords the same—i.e., not removing either `private` or `fileprivate`— and using `semi-` as a placeholder, we want to either have:

  private: surrounding scope
  fileprivate: surrounding file
  semi-internal: surrounding submodule
  internal: surrounding module
  public: all modules (no subclassing)
  open: all modules (with subclassing)

Or:

  private: surrounding scope
  fileprivate: surrounding file
  internal: surrounding submodule
  semi-public: surrounding module
  public: all modules (no subclassing)
  open: all modules (with subclassing)

The difference between the two is that, with `semi-internal` below `internal`, submodule APIs are exposed by default to other submodules; with `semi-public` above `internal`, submodule APIs are encapsulated by default from other submodules.

Either of these would be perfectly workable with the basic design I have proposed. But they both adopt the “flat sea of submodule scopes” perspective. This is far better than what we have now, but it is also not ideal. Truly hierarchical submodule scopes give us a lot more flexibility to structure the implementation of our module.

I think encapsulating by default is the right decision, so we want the `semi-public` design.
But there's also a second reason to use that design: We can anticipate another use case for it. The library resilience design document discusses the idea of "resilience domains"—groups of libraries whose versions are always matched, and which therefore don't need to use resilient representations of each others' data structures—and the idea of having "SPIs", basically APIs that are only public to certain clients. I think these ideas could be conflated, so that a semi-public API would be available both to other submodules in the module and to other libraries in your resilience domain, and that this feature could be used to expose SPIs.

So, that leaves an important question: what the hell do you call this thing? My best suggestions are `confidential` and `privileged`; in the context of information, these are both used to describe information which *is* shared, but only within a select group. (Think, for instance, of attorney-client privilege: You can share this information with your lawyer, but not with anyone else.)

This brings to mind the SE-0025 debate. I can’t say for sure, but it’s quite likely that both of those terms were discussed in that thread. :)

In any case, I agree that identifying a good name for this is worthwhile.

So in short, I suggest adding a single access level to the existing system:

  private
  fileprivate
  internal
  confidential/privileged
  public
  open

This is orthogonal to any other simplification of the access control system, like removing `private` or `fileprivate`.

Appendix A: file system independence

I think we need to decide: Is a translation unit of some sort—whether it's a physical on-disk file or some simulacrum like a database record or just a separate string—something intrinsic to Swift? I think it should be; it simplifies a lot of parts of the language that would otherwise require nesting and explicit scoping.

If translation units are an implicit part of Swift, then this section is not really necessary. If translation units aren't, then we need to rethink a lot of things that are already built in.

I tend to agree. I included this section mostly because Robert and Jaden emphasized file system independence quite heavily in their submodule thread. I wanted to make it very clear that while files play a role in this design it is not coupled to the physical file system at all. Files don’t have any role that isn’t already giving to them by Swift.

I really appreciate all of your feedback Brent! I’m looking forward to continuing the discussion.

···

On Mar 1, 2017, at 12:55 AM, Brent Royal-Gordon <brent@architechies.com> wrote:

On Feb 24, 2017, at 11:34 AM, Matthew Johnson via swift-evolution <swift-evolution@swift.org> wrote:

--
Brent Royal-Gordon
Architechies

Wonderful comments. I really enjoy your take on submodules which keeps most of the power while keeping the simplicity. Comments below:

Sent from my iPhone

Scope-based submodules

   • Proposal: SE-NNNN
   • Authors: Matthew Johnson
   • Review Manager: TBD
   • Status: Awaiting review

Well, this is certainly comprehensive! Sorry about the delay in answering; I've been hosting a house guest and haven't had a lot of free time.

The primary goal of this proposal are to introduce a unit of encapsulation within a module that is larger than a file as a means of adding explicit structure to a large program. All other goals are subordinate to this goal and should be considered in light of it.

I agree with this as the primary goal of a submodule system.

Some other goals of this proposal are:

   • Submodules should help us to manage and understand the internal dependencies of a large, complex system.
   • Submodules should be able to collaborate with peer submodules without necessarily being exposed to the rest of the module.
   • A module should not be required to expose its internal submodule structure to users when symbols are exported.
   • It should be possible to extract a submodule from existing code with minimal friction. The only difficulty should be breaking any circular dependencies.

One goal I don't see mentioned here is "segment the API surface exposed to importing code". The `UIGestureRecognizerSubclass` use case has been thoroughly discussed, but I think there are probably a lot of cases where there are two "sides" to an API and it'd often be helpful to hide one unless it's needed. `URLProtocol` and `URLProtocolClient` come to mind; the many weird little classes and symbols related to `NSAtomicStore` and `NSIncrementalStore` might be another.

(I'm not necessarily suggesting that the Foundation and Core Data overlays should move these into submodules—I'm suggesting that, if they were implemented in a Swift with a submodule feature, they would be candidates for submodule encapsulation.)

Submodule names form a hierarchical path:

   • The fully qualified name of the submodule specified by Submodule.InnerSubmodule is: MyModuleName.Submodule.InnerSubmodule.
   • In this example, InnerSubmodule is a child of Submodule.
   • A submodule may not have the same name as any of its ancestors. This follows the rule used by types.

Does being in a nested submodule have any semantic effect, or is it just a naming trick?

Submodules may not be extended. They form strictly nested scopes.

   • The only way to place code in a submodule is with a submodule declaration at the top of a file.
   • All code in a file exists in a single submodule.

I'm a big supporter of the 1-to-N submodule-to-file approach.

Agreed.

There are several other ways to specify which submodule the top-level scope of a file is in. All of these alternatives share a crucial problem: you can’t tell what submodule your code is in by looking at the file.

The alternatives are:

   • Use a manifest file. This would be painful to maintain.
   • Use file system paths. This is too tightly coupled to physical organization. Appendix A discusses file system independence in more detail.
   • Leave this up to the build system. This makes it more difficult for a module to support multiple build systems.

I'm going to push back on this a little. I don't like the top-of-file `submodule` declaration for several reasons:

   1. Declarations in a Swift file are almost always order-independent. There certainly aren't any that must be the first thing in the file to be valid.

   2. Swift usually keeps configuration stuff out of source files so you can copy and paste snippets of code or whole files around with minimum fuss. Putting `submodule` declarations in files means that developers would need to open and modify those files if they wanted to copy them to a different project. (It's worth noting that your own goal of making it easy to extract submodules into separate modules is undermined by submodule declarations inside files.)

   3. However you're organizing your source code—whether in the file system, an IDE project, or whatever else—it's very likely that you will end up organizing files by submodule. That means either information about submodules will have to be specified twice—once in a canonical declaration and again in source file organization—and kept in sync, or IDEs and tooling will have to interpret the `submodule` declarations in source files and reflect that information in their UIs.

   4. Your cited reason for rejecting build system-based approaches is that "This makes it more difficult for a module to support multiple build systems", but Swift has this same problem in *many* other parts of its design. For instance, module names and dependencies are build system concerns, despite the fact that this makes it harder to support multiple build systems. I can only conclude that supporting multiple build systems with a single code base is, in the long term, a non-goal, presumably by improving the Xcode/SwiftPM story in some way.

I'm still a fan of build-system-based approaches because I think they're better about these issues. The only way that they're worse is that—as you note—it may not be clear which submodule a particular file is in. But I think this is basically a UI problem for editors.

I'm also a big fan of the build-system or file-system approach, for exactly the same reasons. The copy/pasting argument is important IMHO. For example, the suggestion which has been discussed a lot on here to make private change to submodule scope when inside a submodule would have the same negative effect on the ability to copy paste code between a submodule and top-module.

Top-level export

All export statements consist of an access modifier, the export keyword, and a submodule name:

open export ChildSubmodule

Is the access control keyword mandatory?

If we do indeed use `submodule` statements, could we attach the attributes to them, rather than having a separate statement in a different file?

What does it mean if a `public` or `open` symbol is in a submodule which is not `export`ed?

   • A submodule may be published under a different external name using the export as NewName syntax*.

What's the use case for this feature?

   • @implicit causes symbols from the submodule to be implicitly imported when the module is imported.
   • @inline causes the symbols from the submodule to appear as if they had been declared directly within the top-level submodule.

So if you write `@implicit public export Bar` in module `Foo`, then writing `import Foo` also imports `Foo.Bar.Baz` *as* `Foo.Bar.Baz`, whereas `@inline public export Bar` copies `Foo.Bar.Baz` into `Foo`, so it imports as `Foo.Baz`?

What's the use case for supporting both of these behaviors?

Exports within the module

A submodule may bound the maximum visibility of any of its descendent submodules by explicitly exporting it:

I'm not sure how valuable this feature is in this kind of submodule design.

***

To avoid being coy, here's the export control model that *I* think would make the most sense for this general class of submodule system designs:

1. A submodule with `public` or `open` symbols is importable from outside the module. There is no need to separately mark the submodule as importable.

I had not thought of that but it's very elegant.

2. Normally, references within the module to submodule symbols need to be prefixed with the submodule name. (That is, in top-level `Foo` code, you need to write `Bar.Baz` to access `Foo.Bar.Baz`). As a convenience, you can import a submodule, which makes the submodule's symbols available to that file as though they were top-level module symbols.

3. When you import a submodule, you can mark it with `@exported`; this indicates that the symbols in that submodule should be aliased and, if `public` or `open`, re-exported to other modules.

Could you explain this in more detail?

4. There are no special facilities for renaming submodules or implicitly importing submodules.

Importing submodules

Submodules are imported in exactly the same way as an external module by using an import statement.

Okay, but what exactly does importing *do*? Set up un-prefixed private aliases for the submodule's internal-and-up APIs?

There are a few additional details that are not applicable for external modules:

   • Circular imports are not allowed.

Why not? In this design, all submodules are evaluated at once, so I'm not sure why circular imports would be a problem.

// `Grandparent` and all of its descendents can see `Child1` (fully qualified: `Grandparent.Parent.Child1`)
// This reads: `Child1` is scoped to `Grandparent`.

scoped(Grandparent) export Child1

// `Child2` is visible throughout the module but may not be exported for use by clients.
// This reads: `Child2` is scoped to the module.

scoped(module) export Child2

With parameterization, scoped has the power to specify all access levels that Swift has today:

`scoped` == `private` (Swift 3)
`scoped(file)` == `private` (Swift 2 & 4?) == `fileprivate` (Swift 3)
`scoped(submodule)` == `internal`
`scoped(public) scoped(internal, inherit)`* == `public`
`scoped(public)` == `open`

The parameterization of scoped also allows us to reference other scopes that we cannot in today’s system, specifically extensions: scoped(extension) and outer types: scoped(TypeName).

What is the purpose of creating more verbose aliases for existing access levels? I can't think of one, which means that these are redundant.

And if we remove them as redundant, the remaining access control levels look like:

   scoped
   private
   scoped(TypeName)
   internal
   scoped(SomeModule)
   scoped(module)
   scoped(extension)
   public
   open

There's just no logic to the use of the `scoped` keyword here—it doesn't really mean anything other than "we didn't want to assign a keyword to this access level".

***

I think we need to go back to first principles here. The reason to introduce a new access level is that we believe that a submodule is a large enough unit of code that it will simultaneously need to encapsulate some of its implementation details from other submodules, *and* have some of its own implementation details encapsulated from the rest of the submodule.

That's where I disagree. Why would Swift need that level of differentiation and not C# or Java, which have had packages and namespaces without it.

Need is a relative thing. There are many languages that succeed without many of the features Swift has and many languages that succeed partly because of features Swift does not have.

Learning from other languages is a good thing, but in the end the goal is to make Swift the best language it can be. In many cases that happens when we carefully consider our options and find a solution that in some cases is unique.

I am happy that Swift doesn’t follow the example of C# and Java too closely. There are things we can learn from them but also many things we *don’t* want to learn from them. Introducing namespaces without scope boundaries would be a mistake IMO.

···

On Mar 1, 2017, at 1:56 PM, David Hart <david@hartbit.com> wrote:
On 1 Mar 2017, at 07:55, Brent Royal-Gordon via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Feb 24, 2017, at 11:34 AM, Matthew Johnson via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Thus, we need at least three access levels within a submodule: one that exposes an API to other submodules, one that exposes an API throughout a submodule, and one that exposes it to only part of a submodule.

What we do *not* need is a way to allow access only from certain other named submodules. The goal is to separate external and internal interfaces, not to micromanage who can access what.

Basically, that means we need one of two things. Keeping all existing keywords the same—i.e., not removing either `private` or `fileprivate`— and using `semi-` as a placeholder, we want to either have:

   private: surrounding scope
   fileprivate: surrounding file
   semi-internal: surrounding submodule
   internal: surrounding module
   public: all modules (no subclassing)
   open: all modules (with subclassing)

Or:

   private: surrounding scope
   fileprivate: surrounding file
   internal: surrounding submodule
   semi-public: surrounding module
   public: all modules (no subclassing)
   open: all modules (with subclassing)

The difference between the two is that, with `semi-internal` below `internal`, submodule APIs are exposed by default to other submodules; with `semi-public` above `internal`, submodule APIs are encapsulated by default from other submodules.

I think encapsulating by default is the right decision, so we want the `semi-public` design. But there's also a second reason to use that design: We can anticipate another use case for it. The library resilience design document discusses the idea of "resilience domains"—groups of libraries whose versions are always matched, and which therefore don't need to use resilient representations of each others' data structures—and the idea of having "SPIs", basically APIs that are only public to certain clients. I think these ideas could be conflated, so that a semi-public API would be available both to other submodules in the module and to other libraries in your resilience domain, and that this feature could be used to expose SPIs.

So, that leaves an important question: what the hell do you call this thing? My best suggestions are `confidential` and `privileged`; in the context of information, these are both used to describe information which *is* shared, but only within a select group. (Think, for instance, of attorney-client privilege: You can share this information with your lawyer, but not with anyone else.)

So in short, I suggest adding a single access level to the existing system:

   private
   fileprivate
   internal
   confidential/privileged
   public
   open

This is orthogonal to any other simplification of the access control system, like removing `private` or `fileprivate`.

Yes, I would still like to see scope private go away.

Appendix A: file system independence

I think we need to decide: Is a translation unit of some sort—whether it's a physical on-disk file or some simulacrum like a database record or just a separate string—something intrinsic to Swift? I think it should be; it simplifies a lot of parts of the language that would otherwise require nesting and explicit scoping.

If translation units are an implicit part of Swift, then this section is not really necessary. If translation units aren't, then we need to rethink a lot of things that are already built in.

--
Brent Royal-Gordon
Architechies

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

2. Normally, references within the module to submodule symbols need to be prefixed with the submodule name. (That is, in top-level `Foo` code, you need to write `Bar.Baz` to access `Foo.Bar.Baz`). As a convenience, you can import a submodule, which makes the submodule's symbols available to that file as though they were top-level module symbols.

3. When you import a submodule, you can mark it with `@exported`; this indicates that the symbols in that submodule should be aliased and, if `public` or `open`, re-exported to other modules.

Could you explain this in more detail?

Let's say you're writing the `WorldOptimization` module, and you have a `SpimsterWicket` submodule with a public `WicketShell` type. In the main `WorldOptimization` module, you could write:

  class Optimizer: Person {
    var wicketShell: SpimsterWicket.WicketShell?
    …
  }

But if you don't want to write out the name of the `SpimsterWicket` submodule, you could import it:

  import WorldOptimization.SpimsterWicket
  
  class Optimizer: Person {
    var wicketShell: WicketShell?
    …
  }

Outside the `WorldOptimization` module, of course, you'd still need to write `import WorldOptimization.SpimsterWicket` to get access to the `WicketShell` type. (Or perhaps importing `WorldOptimization` would allow you to access it as `SpimsterWicket.WicketShell`--it's an open question.) On the other hand, if `WorldOptimization` instead included:

  @exported import WorldOptimization.SpimsterWicket
  
  class Optimizer: Person {
    var wicketShell: WicketShell?
    …
  }

Then anyone who imported `WorldOptimization` could access `WicketShell` directly, as though they too had explicitly imported `WorldOptimization.SpimsterWicket`.

I think we need to go back to first principles here. The reason to introduce a new access level is that we believe that a submodule is a large enough unit of code that it will simultaneously need to encapsulate some of its implementation details from other submodules, *and* have some of its own implementation details encapsulated from the rest of the submodule.

That's where I disagree. Why would Swift need that level of differentiation and not C# or Java, which have had packages and namespaces without it.

C# has friend assemblies <Microsoft Learn: Build skills that open doors in your career, PublisherIdentityPermission <c# - Restricting the use of an assembly from assemblies that are not signed - Stack Overflow, and similar security mechanisms to restrict access to certain APIs. I have not been able to locate similar features in Java; instead, people seem to recommend circuitous design patterns to make up for it: <APIDesignPatterns:FriendPackages&gt - APIDesign;

More broadly, though? Swift is trying to reach to lower levels than C# or (particularly) Java usually do. If we're serious about Swift as a systems language, and we're also serious about the kind of careful layering that Apple's OSes are known for, we need some way to expose SPIs to higher layers. And it'd be good if that were built into the language.

···

On Mar 1, 2017, at 11:56 AM, David Hart <david@hartbit.com> wrote:

--
Brent Royal-Gordon
Architechies

2. Normally, references within the module to submodule symbols need to be prefixed with the submodule name. (That is, in top-level `Foo` code, you need to write `Bar.Baz` to access `Foo.Bar.Baz`). As a convenience, you can import a submodule, which makes the submodule's symbols available to that file as though they were top-level module symbols.

3. When you import a submodule, you can mark it with `@exported`; this indicates that the symbols in that submodule should be aliased and, if `public` or `open`, re-exported to other modules.

Could you explain this in more detail?

Let's say you're writing the `WorldOptimization` module, and you have a `SpimsterWicket` submodule with a public `WicketShell` type. In the main `WorldOptimization` module, you could write:

  class Optimizer: Person {
    var wicketShell: SpimsterWicket.WicketShell?
    …
  }

But if you don't want to write out the name of the `SpimsterWicket` submodule, you could import it:

  import WorldOptimization.SpimsterWicket
  
  class Optimizer: Person {
    var wicketShell: WicketShell?
    …
  }

Outside the `WorldOptimization` module, of course, you'd still need to write `import WorldOptimization.SpimsterWicket` to get access to the `WicketShell` type. (Or perhaps importing `WorldOptimization` would allow you to access it as `SpimsterWicket.WicketShell`--it's an open question.) On the other hand, if `WorldOptimization` instead included:

  @exported import WorldOptimization.SpimsterWicket
  
  class Optimizer: Person {
    var wicketShell: WicketShell?
    …
  }

Then anyone who imported `WorldOptimization` could access `WicketShell` directly, as though they too had explicitly imported `WorldOptimization.SpimsterWicket`.

If I understand correctly, this does not change the fully qualified name seen by users of the module, only sets up an implicit import of the submodule when the module is imported. Is that correct?

···

On Mar 2, 2017, at 10:08 PM, Brent Royal-Gordon <brent@architechies.com> wrote:

On Mar 1, 2017, at 11:56 AM, David Hart <david@hartbit.com> wrote:

I think we need to go back to first principles here. The reason to introduce a new access level is that we believe that a submodule is a large enough unit of code that it will simultaneously need to encapsulate some of its implementation details from other submodules, *and* have some of its own implementation details encapsulated from the rest of the submodule.

That's where I disagree. Why would Swift need that level of differentiation and not C# or Java, which have had packages and namespaces without it.

C# has friend assemblies <Microsoft Learn: Build skills that open doors in your career, PublisherIdentityPermission <c# - Restricting the use of an assembly from assemblies that are not signed - Stack Overflow, and similar security mechanisms to restrict access to certain APIs. I have not been able to locate similar features in Java; instead, people seem to recommend circuitous design patterns to make up for it: <APIDesignPatterns:FriendPackages&gt - APIDesign;

More broadly, though? Swift is trying to reach to lower levels than C# or (particularly) Java usually do. If we're serious about Swift as a systems language, and we're also serious about the kind of careful layering that Apple's OSes are known for, we need some way to expose SPIs to higher layers. And it'd be good if that were built into the language.

--
Brent Royal-Gordon
Architechies

Right.

···

On Mar 3, 2017, at 12:56 PM, Matthew Johnson <matthew@anandabits.com> wrote:

Outside the `WorldOptimization` module, of course, you'd still need to write `import WorldOptimization.SpimsterWicket` to get access to the `WicketShell` type. (Or perhaps importing `WorldOptimization` would allow you to access it as `SpimsterWicket.WicketShell`--it's an open question.) On the other hand, if `WorldOptimization` instead included:

  @exported import WorldOptimization.SpimsterWicket
  
  class Optimizer: Person {
    var wicketShell: WicketShell?
    …
  }

Then anyone who imported `WorldOptimization` could access `WicketShell` directly, as though they too had explicitly imported `WorldOptimization.SpimsterWicket`.

If I understand correctly, this does not change the fully qualified name seen by users of the module, only sets up an implicit import of the submodule when the module is imported. Is that correct?

--
Brent Royal-Gordon
Architechies