Yet Another Take on Swift Sub-modules

Evolution Discussion
1

I’ve read through the last couple of Swift (sub)Module proposals put forward, and since my particular use-cases for a sub-module solution seemed to be under-served by them, I’ve decided to write up my thoughts on the matter to prompt discussion.

Perhaps my use-cases are outliers, and my approach will be deemed naive by the community… I’m happy to learn better ways of doing things in Swift, and welcome any thoughts, criticism, or illumination related to these ideas.

I’m including the write-up below, but it’s also available as a gist: Yet Another Take on Swift Sub-modules · GitHub

···

—

# Sub-modules

A sub-module solution in Swift should have the following properties:

* Extremely light-weight
* Low API surface area
* Adopt progressive disclosure
* Integrate with Access Control features to enable a level of encapsulation & hiding between the Module and File level
* Be permeable when desired

## Discussion

As we get deeper into building real applications & frameworks with Swift, we begin to realize that having a way to express relationships between types is desireable. Currently, Swift only allows us to express these relationships at two levels, the Module and the File.

The Module boundary is acceptable for small, focused frameworks, while the File boundary is acceptable for small, focused Types, but both levels can be unweildy when dealing with certain cases where a cluster of internally related types needs to know about each other but may only want to publish a narrow set of APIs to the surrounding code, or in large complex applications which are necessarily structured as a single Module. In these cases, we wind up with large monolithic Modules or (even worse) large monolithic Files.

I have seen this proliferation of Huge Sprawling Files (HSFs) in my own code, and seek a way to combat this rising tide.

## Goals

It is a goal of this proposal to:

* Suggest a mechanism for organizing code between the Module and File levels that is as lightweight and low-friction as possible
* Provide mechanisms for authors to create both "hard" and "soft" API boundaries between the Module and File levels of their code

## Anti-Goals

It is not a goal of this proposal to:

* Move Swift away from filesystem-based organization
* Significantly alter the current Access Control philosophy of Swift

## Proposal Notes

Please take the following proposal wholely as a Straw-Man... I would be equally satisfied with any solution which meets the critera described at the top of this document.

Unless specified otherwise, all spellings proposed below are to be considered straw-men, and merely illustrative of the concepts.

## Proposed Solution

Two things are clear to me after using Swift and following the Swift Evolution list since their respective publications:

1. Swift has a preference for file-based organization
2. Vocal Swift Users dislike `fileprivate` and want to revert to Swift2-style `private`

Because of #1, this proposal does not seek to change Swift's inherent file-system organization, and instead will expand on it.

Since I personally fall into the camp described by #2, and most of the community response to this has been "Lets wait to deal with that until sub-modules", I'm making this proposal assuming that solving that quagmire is in-scope for this propsoal.

### Changes to Access Control Modifiers

As part of this proposal, I suggest the following changes to Swift 3's Access Control modifiers:

* Revert `private` to Swift 2's meaning: "hidden outside the file"
* Remove `fileprivate` as redundant

This is potentially a source-breaking change. However, it is interesting to note that this change is **not** required for the following proposal to function.

Changes that *are* necessary are:

* Change the spelling of `internal` to `module` (making `module` the new default)
* Introduce a new modifier `internal` to mean "Internal to the current sub-module and its child-sub-modules"

These changes are *not* source-breaking because the new `internal` modifier acts exactly as the old `internal` modifier unless it is used within a sub-module. The specific spelling of this new `internal` modifier is necessary to maintain backwards source compatibility.

The new `module` modifier allows authors to make APIs permeable between sub-modules while still hidden outside the owning Module if desired.

All other Access Control modifiers behave the same as they currently do irrespective of sub-module boundaries, so:

* `public` => Visible outside the Module
* `open` => Sub-classable outside the Module

### Making a Sub-module

To create a sub-module within a Module (or sub-module) is simple: The author creates a directory, and places a "sub-module declaration file" within the directory:

//  __submodule.swift
//  MyModule

submodule SubA

Then any files within that directory are part of the sub-module:

//  Foo.swift
//  MyModule.SubA

struct Foo {
    private var mine: Bool
    internal var sub: Bool
    module var mod: Bool
}

public struct Bar {
    module var mod: Bool
    public var pub: Bool
}

This creates a sub-module called "SubA" within the module "MyModule". All files within the directory in which this file appears are understood to be contained by this sub-module.

If in the future we choose to add additional complexity (versioning, #availability, etc) to the sub-module syntax, the sub-module declaration gives a natural home for this configuration.

It's important to note some benefits of this approach:

* Using the "special file" means that not all Directories are automatically submodules
* Any given source file may only be a member of 1 submodule at a time
* Use of filesystem structure to denote sub-modules plays nicely with source control
* The sub-module structure is instantly clear whether using an IDE (which can be taught to parse the `__submodule.swift` files to decorate the UI), or simple text-editor (assuming a convention of naming the Directory the same as the sub-module, which is a linter problem)

### Using Sub-modules

Referencing a sub-module should be natural and clear at this point:

#### From Within the Parent Module/Sub-module

Sub-modules are simply code-organization & namespacing tools within modules. As such, when referenced from within their parent Module, there is no need for `import`s

//  in MyModule

let foo = SubA.Foo()
foo.mine = true // Compiler error because it's private
foo.sub = true  // Compiler error because it's internal to the sub-module
foo.mod = true  // OK

#### From Outside the Parent Module/Sub-module

When referenced from outside their parent Module, one imports the whole module in the standard way:

import MyModule

let foo = SubA.Foo() // Compiler error because it's internal to the Module

let bar = SubA.Bar() // OK
bar.mod = true  // Compiler error because it's internal to the Module
bar.pub = true  // OK

## What this Proposal Deliberately Omits

This proposal deliberately omits several concepts which may be integral to various use-cases for sub-modules, primarily because they can be treated as purely additive concepts and I don't wish to weigh down the consideration of the overall approach with a larger API surface area that might be debated separately. I.e: Keep it as small as possible for now, then if it's any good, iterate on the design.

### Inter-Sub-Module Access Control

One might ask given a sub-module structure like:

MyModule
  >
  +--- SubA
        >
        +--- SubB

"How can SubB hide properties from MyModule without hiding them from SubA?"

This is a valid question, and not answered by this proposal for two reasons:

* This trivial case could be solved by simply adding a new modifier `submodule` if we so desired, but:
* In the absence of any direct response, the status-quo provides a work-around: Omit the sub-sub-module structure and use the file-access constraints of `private`
* This overall problem probably should be solved by addressing larger questions in the Access Control scheme of Swift, irrespective of the sub-module mechanism

### Expressiveness of Sub-module Imports

One might ask: "Why can't I import only a specific sub-module or alias a sub-module?"

I have ignored this aspect of submodules because the question of `import` expressiveness is a separate issue in my mind. The fact that we cannot say:

import MyModule as Foo

Has no relationship to the lack of sub-modules in Swift.

If the community deems it an important enough use-case to warrant altering import behavior, so be it, but that can be treated as purely additive to this proposal.

But it should be understood that this approach to sub-modules is not designed to provide an expressive "exports" capability. It is primarily interested in organizing code *within* a Module

2

I’ve read through the last couple of Swift (sub)Module proposals put forward, and since my particular use-cases for a sub-module solution seemed to be under-served by them, I’ve decided to write up my thoughts on the matter to prompt discussion.

Perhaps my use-cases are outliers, and my approach will be deemed naive by the community… I’m happy to learn better ways of doing things in Swift, and welcome any thoughts, criticism, or illumination related to these ideas.

I’m including the write-up below, but it’s also available as a gist: Yet Another Take on Swift Sub-modules · GitHub

—

# Sub-modules

A sub-module solution in Swift should have the following properties:

* Extremely light-weight
* Low API surface area
* Adopt progressive disclosure
* Integrate with Access Control features to enable a level of encapsulation & hiding between the Module and File level
* Be permeable when desired

## Discussion

As we get deeper into building real applications & frameworks with Swift, we begin to realize that having a way to express relationships between types is desireable. Currently, Swift only allows us to express these relationships at two levels, the Module and the File.

The Module boundary is acceptable for small, focused frameworks, while the File boundary is acceptable for small, focused Types, but both levels can be unweildy when dealing with certain cases where a cluster of internally related types needs to know about each other but may only want to publish a narrow set of APIs to the surrounding code, or in large complex applications which are necessarily structured as a single Module. In these cases, we wind up with large monolithic Modules or (even worse) large monolithic Files.

I have seen this proliferation of Huge Sprawling Files (HSFs) in my own code, and seek a way to combat this rising tide.

## Goals

It is a goal of this proposal to:

* Suggest a mechanism for organizing code between the Module and File levels that is as lightweight and low-friction as possible
* Provide mechanisms for authors to create both "hard" and "soft" API boundaries between the Module and File levels of their code

## Anti-Goals

It is not a goal of this proposal to:

* Move Swift away from filesystem-based organization
* Significantly alter the current Access Control philosophy of Swift

## Proposal Notes

Please take the following proposal wholely as a Straw-Man... I would be equally satisfied with any solution which meets the critera described at the top of this document.

Unless specified otherwise, all spellings proposed below are to be considered straw-men, and merely illustrative of the concepts.

## Proposed Solution

Two things are clear to me after using Swift and following the Swift Evolution list since their respective publications:

1. Swift has a preference for file-based organization
2. Vocal Swift Users dislike `fileprivate` and want to revert to Swift2-style `private`

Because of #1, this proposal does not seek to change Swift's inherent file-system organization, and instead will expand on it.

Since I personally fall into the camp described by #2, and most of the community response to this has been "Lets wait to deal with that until sub-modules", I'm making this proposal assuming that solving that quagmire is in-scope for this propsoal.

### Changes to Access Control Modifiers

As part of this proposal, I suggest the following changes to Swift 3's Access Control modifiers:

* Revert `private` to Swift 2's meaning: "hidden outside the file"
* Remove `fileprivate` as redundant

This is potentially a source-breaking change. However, it is interesting to note that this change is **not** required for the following proposal to function.

Changes that *are* necessary are:

* Change the spelling of `internal` to `module` (making `module` the new default)
* Introduce a new modifier `internal` to mean "Internal to the current sub-module and its child-sub-modules”

Can you give concrete examples of use cases where a descendent submodule needs access to symbols declared by an ancestor? I gave some thought to this when drafting my proposal and came to the conclusion that this runs against the grain of layering and is likely to be a bad idea in practice. If there are use cases I didn’t consider I am very interested in learning about them.

These changes are *not* source-breaking because the new `internal` modifier acts exactly as the old `internal` modifier unless it is used within a sub-module. The specific spelling of this new `internal` modifier is necessary to maintain backwards source compatibility.

The new `module` modifier allows authors to make APIs permeable between sub-modules while still hidden outside the owning Module if desired.

All other Access Control modifiers behave the same as they currently do irrespective of sub-module boundaries, so:

* `public` => Visible outside the Module
* `open` => Sub-classable outside the Module

### Making a Sub-module

To create a sub-module within a Module (or sub-module) is simple: The author creates a directory, and places a "sub-module declaration file" within the directory:

//  __submodule.swift

Why the double underscore prefix? To make it sort to the top in a file browser?

Is this file allowed to have any Swift code? Or is it limited to submodule-related declarations only? If the latter, why not use an extension such as .submodule or .swiftmodule to differentiate it from ordinary Swift files and allow the submodule to be named by the name of this file?

// MyModule

submodule SubA


Then any files within that directory are part of the sub\-module:

// Foo.swift
// MyModule.SubA

struct Foo {
private var mine: Bool
internal var sub: Bool
module var mod: Bool
}

public struct Bar {
module var mod: Bool
public var pub: Bool
}


This creates a sub\-module called "SubA" within the module "MyModule"\. All files within the directory in which this file appears are understood to be contained by this sub\-module\.

If in the future we choose to add additional complexity \(versioning, \#availability, etc\) to the sub\-module syntax, the sub\-module declaration gives a natural home for this configuration\.

It's important to note some benefits of this approach:

\* Using the "special file" means that not all Directories are automatically submodules
\* Any given source file may only be a member of 1 submodule at a time
\* Use of filesystem structure to denote sub\-modules plays nicely with source control
\* The sub\-module structure is instantly clear whether using an IDE \(which can be taught to parse the \`\_\_submodule\.swift\` files to decorate the UI\), or simple text\-editor \(assuming a convention of naming the Directory the same as the sub\-module, which is a linter problem\)

If we’re going to use the file system to organize submodules this seems like a reasonable approach. It allows larger submodules to have folder hierarchies within them and also creates a central location for submodule-related declarations.

A primary flaw I see in this approach is that Xcode is the dominant IDE for Swift and the way Xcode handles files is not conducive to file-system organization. I really detest the way Xcode handles this and would vastly prefer that it simply reflected the physical file system hierarchy but I don’t think that will change any time soon. On the other hand maybe a file system based submodule system in Swift would motivate the Xcode team to better reflect the physical file system organization.

···

On Mar 3, 2017, at 9:24 AM, Karim Nassar via swift-evolution <swift-evolution@swift.org> wrote:

### Using Sub-modules

Referencing a sub-module should be natural and clear at this point:

#### From Within the Parent Module/Sub-module

Sub-modules are simply code-organization & namespacing tools within modules. As such, when referenced from within their parent Module, there is no need for `import`s

//  in MyModule

let foo = SubA.Foo()
foo.mine = true // Compiler error because it's private
foo.sub = true  // Compiler error because it's internal to the sub-module
foo.mod = true  // OK

#### From Outside the Parent Module/Sub-module

When referenced from outside their parent Module, one imports the whole module in the standard way:

import MyModule

let foo = SubA.Foo() // Compiler error because it's internal to the Module

let bar = SubA.Bar() // OK
bar.mod = true  // Compiler error because it's internal to the Module
bar.pub = true  // OK

## What this Proposal Deliberately Omits

This proposal deliberately omits several concepts which may be integral to various use-cases for sub-modules, primarily because they can be treated as purely additive concepts and I don't wish to weigh down the consideration of the overall approach with a larger API surface area that might be debated separately. I.e: Keep it as small as possible for now, then if it's any good, iterate on the design.

### Inter-Sub-Module Access Control

One might ask given a sub-module structure like:

MyModule
 >
 +--- SubA
       >
       +--- SubB

"How can SubB hide properties from MyModule without hiding them from SubA?"

This is a valid question, and not answered by this proposal for two reasons:

* This trivial case could be solved by simply adding a new modifier `submodule` if we so desired, but:
* In the absence of any direct response, the status-quo provides a work-around: Omit the sub-sub-module structure and use the file-access constraints of `private`
* This overall problem probably should be solved by addressing larger questions in the Access Control scheme of Swift, irrespective of the sub-module mechanism

### Expressiveness of Sub-module Imports

One might ask: "Why can't I import only a specific sub-module or alias a sub-module?"

I have ignored this aspect of submodules because the question of `import` expressiveness is a separate issue in my mind. The fact that we cannot say:

import MyModule as Foo

Has no relationship to the lack of sub-modules in Swift.

If the community deems it an important enough use-case to warrant altering import behavior, so be it, but that can be treated as purely additive to this proposal.

But it should be understood that this approach to sub-modules is not designed to provide an expressive "exports" capability. It is primarily interested in organizing code *within* a Module

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

3

I don’t like the file location based approach taken, see inline.

Rien.

I’ve read through the last couple of Swift (sub)Module proposals put forward, and since my particular use-cases for a sub-module solution seemed to be under-served by them, I’ve decided to write up my thoughts on the matter to prompt discussion.

Perhaps my use-cases are outliers, and my approach will be deemed naive by the community… I’m happy to learn better ways of doing things in Swift, and welcome any thoughts, criticism, or illumination related to these ideas.

I’m including the write-up below, but it’s also available as a gist: Yet Another Take on Swift Sub-modules · GitHub

—

# Sub-modules

A sub-module solution in Swift should have the following properties:

* Extremely light-weight
* Low API surface area
* Adopt progressive disclosure
* Integrate with Access Control features to enable a level of encapsulation & hiding between the Module and File level
* Be permeable when desired

## Discussion

As we get deeper into building real applications & frameworks with Swift, we begin to realize that having a way to express relationships between types is desireable. Currently, Swift only allows us to express these relationships at two levels, the Module and the File.

The Module boundary is acceptable for small, focused frameworks, while the File boundary is acceptable for small, focused Types, but both levels can be unweildy when dealing with certain cases where a cluster of internally related types needs to know about each other but may only want to publish a narrow set of APIs to the surrounding code, or in large complex applications which are necessarily structured as a single Module. In these cases, we wind up with large monolithic Modules or (even worse) large monolithic Files.

I have seen this proliferation of Huge Sprawling Files (HSFs) in my own code, and seek a way to combat this rising tide.

## Goals

It is a goal of this proposal to:

* Suggest a mechanism for organizing code between the Module and File levels that is as lightweight and low-friction as possible
* Provide mechanisms for authors to create both "hard" and "soft" API boundaries between the Module and File levels of their code

## Anti-Goals

It is not a goal of this proposal to:

* Move Swift away from filesystem-based organization
* Significantly alter the current Access Control philosophy of Swift

## Proposal Notes

Please take the following proposal wholely as a Straw-Man... I would be equally satisfied with any solution which meets the critera described at the top of this document.

Unless specified otherwise, all spellings proposed below are to be considered straw-men, and merely illustrative of the concepts.

## Proposed Solution

Two things are clear to me after using Swift and following the Swift Evolution list since their respective publications:

1. Swift has a preference for file-based organization
2. Vocal Swift Users dislike `fileprivate` and want to revert to Swift2-style `private`

Because of #1, this proposal does not seek to change Swift's inherent file-system organization, and instead will expand on it.

Since I personally fall into the camp described by #2, and most of the community response to this has been "Lets wait to deal with that until sub-modules", I'm making this proposal assuming that solving that quagmire is in-scope for this propsoal.

### Changes to Access Control Modifiers

As part of this proposal, I suggest the following changes to Swift 3's Access Control modifiers:

* Revert `private` to Swift 2's meaning: "hidden outside the file"
* Remove `fileprivate` as redundant

This is potentially a source-breaking change. However, it is interesting to note that this change is **not** required for the following proposal to function.

Changes that *are* necessary are:

* Change the spelling of `internal` to `module` (making `module` the new default)
* Introduce a new modifier `internal` to mean "Internal to the current sub-module and its child-sub-modules"

These changes are *not* source-breaking because the new `internal` modifier acts exactly as the old `internal` modifier unless it is used within a sub-module. The specific spelling of this new `internal` modifier is necessary to maintain backwards source compatibility.

The new `module` modifier allows authors to make APIs permeable between sub-modules while still hidden outside the owning Module if desired.

All other Access Control modifiers behave the same as they currently do irrespective of sub-module boundaries, so:

* `public` => Visible outside the Module
* `open` => Sub-classable outside the Module

### Making a Sub-module

To create a sub-module within a Module (or sub-module) is simple: The author creates a directory, and places a "sub-module declaration file" within the directory:

I don’t like “behind the scenes” magic. Any file should stand on its own. i.e. I want to see in the file itself what it can do, and what it uses from other files. Everything should be defined in the file itself.

Its place in the file system should be completely irrelevant.

//  __submodule.swift
//  MyModule

submodule SubA

Then any files within that directory are part of the sub-module:

//  Foo.swift
//  MyModule.SubA

struct Foo {
   private var mine: Bool
   internal var sub: Bool
   module var mod: Bool
}

public struct Bar {
   module var mod: Bool
   public var pub: Bool
}

This creates a sub-module called "SubA" within the module "MyModule". All files within the directory in which this file appears are understood to be contained by this sub-module.

If in the future we choose to add additional complexity (versioning, availability, etc) to the sub-module syntax, the sub-module declaration gives a natural home for this configuration.

It's important to note some benefits of this approach:

* Using the "special file" means that not all Directories are automatically submodules
* Any given source file may only be a member of 1 submodule at a time
* Use of filesystem structure to denote sub-modules plays nicely with source control
* The sub-module structure is instantly clear whether using an IDE (which can be taught to parse the `__submodule.swift` files to decorate the UI), or simple text-editor (assuming a convention of naming the Directory the same as the sub-module, which is a linter problem)

### Using Sub-modules

Referencing a sub-module should be natural and clear at this point:

#### From Within the Parent Module/Sub-module

Sub-modules are simply code-organization & namespacing tools within modules. As such, when referenced from within their parent Module, there is no need for `import`s

When I read code, the location or source of each item should be clearly traceable. Either by lexical structure or by (path) file name. If I can access the contents of another file based on the location of that other file in the file system… nope that will generate problems due to “missing file” etc.

Besides I think this could easily become a nightmare for autocomplete in IDEs ?

···

On 03 Mar 2017, at 16:24, Karim Nassar via swift-evolution <swift-evolution@swift.org> wrote:

//  in MyModule

let foo = SubA.Foo()
foo.mine = true // Compiler error because it's private
foo.sub = true  // Compiler error because it's internal to the sub-module
foo.mod = true  // OK

#### From Outside the Parent Module/Sub-module

When referenced from outside their parent Module, one imports the whole module in the standard way:

import MyModule

let foo = SubA.Foo() // Compiler error because it's internal to the Module

let bar = SubA.Bar() // OK
bar.mod = true  // Compiler error because it's internal to the Module
bar.pub = true  // OK

## What this Proposal Deliberately Omits

This proposal deliberately omits several concepts which may be integral to various use-cases for sub-modules, primarily because they can be treated as purely additive concepts and I don't wish to weigh down the consideration of the overall approach with a larger API surface area that might be debated separately. I.e: Keep it as small as possible for now, then if it's any good, iterate on the design.

### Inter-Sub-Module Access Control

One might ask given a sub-module structure like:

MyModule
 >
 +--- SubA
       >
       +--- SubB

"How can SubB hide properties from MyModule without hiding them from SubA?"

This is a valid question, and not answered by this proposal for two reasons:

* This trivial case could be solved by simply adding a new modifier `submodule` if we so desired, but:
* In the absence of any direct response, the status-quo provides a work-around: Omit the sub-sub-module structure and use the file-access constraints of `private`
* This overall problem probably should be solved by addressing larger questions in the Access Control scheme of Swift, irrespective of the sub-module mechanism

### Expressiveness of Sub-module Imports

One might ask: "Why can't I import only a specific sub-module or alias a sub-module?"

I have ignored this aspect of submodules because the question of `import` expressiveness is a separate issue in my mind. The fact that we cannot say:

import MyModule as Foo

Has no relationship to the lack of sub-modules in Swift.

If the community deems it an important enough use-case to warrant altering import behavior, so be it, but that can be treated as purely additive to this proposal.

But it should be understood that this approach to sub-modules is not designed to provide an expressive "exports" capability. It is primarily interested in organizing code *within* a Module

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

4

I strongly agree with all your points. The other submodule proposals are too complex for me so I'm very happy to see a different solution which more closely fits my design priorities.

Comments inline:

I’ve read through the last couple of Swift (sub)Module proposals put forward, and since my particular use-cases for a sub-module solution seemed to be under-served by them, I’ve decided to write up my thoughts on the matter to prompt discussion.

Perhaps my use-cases are outliers, and my approach will be deemed naive by the community… I’m happy to learn better ways of doing things in Swift, and welcome any thoughts, criticism, or illumination related to these ideas.

I don't think your use cases are outliers. I think your solution really feels more Swifty by starting simple.

I’m including the write-up below, but it’s also available as a gist: Yet Another Take on Swift Sub-modules · GitHub

—

# Sub-modules

A sub-module solution in Swift should have the following properties:

* Extremely light-weight
* Low API surface area
* Adopt progressive disclosure
* Integrate with Access Control features to enable a level of encapsulation & hiding between the Module and File level
* Be permeable when desired

Yes, yes, yes. Very good goals :)

## Discussion

As we get deeper into building real applications & frameworks with Swift, we begin to realize that having a way to express relationships between types is desireable. Currently, Swift only allows us to express these relationships at two levels, the Module and the File.

The Module boundary is acceptable for small, focused frameworks, while the File boundary is acceptable for small, focused Types, but both levels can be unweildy when dealing with certain cases where a cluster of internally related types needs to know about each other but may only want to publish a narrow set of APIs to the surrounding code, or in large complex applications which are necessarily structured as a single Module. In these cases, we wind up with large monolithic Modules or (even worse) large monolithic Files.

I have seen this proliferation of Huge Sprawling Files (HSFs) in my own code, and seek a way to combat this rising tide.

## Goals

It is a goal of this proposal to:

* Suggest a mechanism for organizing code between the Module and File levels that is as lightweight and low-friction as possible
* Provide mechanisms for authors to create both "hard" and "soft" API boundaries between the Module and File levels of their code

## Anti-Goals

It is not a goal of this proposal to:

* Move Swift away from filesystem-based organization
* Significantly alter the current Access Control philosophy of Swift

## Proposal Notes

Please take the following proposal wholely as a Straw-Man... I would be equally satisfied with any solution which meets the critera described at the top of this document.

Unless specified otherwise, all spellings proposed below are to be considered straw-men, and merely illustrative of the concepts.

## Proposed Solution

Two things are clear to me after using Swift and following the Swift Evolution list since their respective publications:

1. Swift has a preference for file-based organization
2. Vocal Swift Users dislike `fileprivate` and want to revert to Swift2-style `private`

Because of #1, this proposal does not seek to change Swift's inherent file-system organization, and instead will expand on it.

Since I personally fall into the camp described by #2, and most of the community response to this has been "Lets wait to deal with that until sub-modules", I'm making this proposal assuming that solving that quagmire is in-scope for this propsoal.

I have posted a proposal which proposes #2. It should hopefully get merged and reviewed soon.

### Changes to Access Control Modifiers

As part of this proposal, I suggest the following changes to Swift 3's Access Control modifiers:

* Revert `private` to Swift 2's meaning: "hidden outside the file"
* Remove `fileprivate` as redundant

This is potentially a source-breaking change. However, it is interesting to note that this change is **not** required for the following proposal to function.

Changes that *are* necessary are:

* Change the spelling of `internal` to `module` (making `module` the new default)
* Introduce a new modifier `internal` to mean "Internal to the current sub-module and its child-sub-modules"

I like that you introduce only one access level and maintain source compatibility. This is also what makes the proposal fit the "progressive disclosure" goal: you don't need to learn or worry about the `module` access level until you start working with submodules.

These changes are *not* source-breaking because the new `internal` modifier acts exactly as the old `internal` modifier unless it is used within a sub-module. The specific spelling of this new `internal` modifier is necessary to maintain backwards source compatibility.

The new `module` modifier allows authors to make APIs permeable between sub-modules while still hidden outside the owning Module if desired.

All other Access Control modifiers behave the same as they currently do irrespective of sub-module boundaries, so:

* `public` => Visible outside the Module
* `open` => Sub-classable outside the Module

### Making a Sub-module

To create a sub-module within a Module (or sub-module) is simple: The author creates a directory, and places a "sub-module declaration file" within the directory:

//  __submodule.swift
//  MyModule

submodule SubA

Like the previous response, I think this potentially the weakest part of he proposal. I like the idea of a file-system based submodule system but I'm not sure I like this version of it. People are going to be very vocal about this too. Perhaps you could rewrite the proposal with several solutions for this part? Let people discuss the different solutions.

Then any files within that directory are part of the sub-module:

//  Foo.swift
//  MyModule.SubA

struct Foo {
   private var mine: Bool
   internal var sub: Bool
   module var mod: Bool
}

public struct Bar {
   module var mod: Bool
   public var pub: Bool
}

This creates a sub-module called "SubA" within the module "MyModule". All files within the directory in which this file appears are understood to be contained by this sub-module.

If in the future we choose to add additional complexity (versioning, availability, etc) to the sub-module syntax, the sub-module declaration gives a natural home for this configuration.

It's important to note some benefits of this approach:

* Using the "special file" means that not all Directories are automatically submodules
* Any given source file may only be a member of 1 submodule at a time
* Use of filesystem structure to denote sub-modules plays nicely with source control
* The sub-module structure is instantly clear whether using an IDE (which can be taught to parse the `__submodule.swift` files to decorate the UI), or simple text-editor (assuming a convention of naming the Directory the same as the sub-module, which is a linter problem)

### Using Sub-modules

Referencing a sub-module should be natural and clear at this point:

#### From Within the Parent Module/Sub-module

Sub-modules are simply code-organization & namespacing tools within modules. As such, when referenced from within their parent Module, there is no need for `import`s

//  in MyModule

let foo = SubA.Foo()
foo.mine = true // Compiler error because it's private
foo.sub = true  // Compiler error because it's internal to the sub-module
foo.mod = true  // OK

#### From Outside the Parent Module/Sub-module

When referenced from outside their parent Module, one imports the whole module in the standard way:

import MyModule

let foo = SubA.Foo() // Compiler error because it's internal to the Module

let bar = SubA.Bar() // OK
bar.mod = true  // Compiler error because it's internal to the Module
bar.pub = true  // OK

## What this Proposal Deliberately Omits

This proposal deliberately omits several concepts which may be integral to various use-cases for sub-modules, primarily because they can be treated as purely additive concepts and I don't wish to weigh down the consideration of the overall approach with a larger API surface area that might be debated separately. I.e: Keep it as small as possible for now, then if it's any good, iterate on the design.

### Inter-Sub-Module Access Control

One might ask given a sub-module structure like:

MyModule
 >
 +--- SubA
       >
       +--- SubB

"How can SubB hide properties from MyModule without hiding them from SubA?"

This is a valid question, and not answered by this proposal for two reasons:

* This trivial case could be solved by simply adding a new modifier `submodule` if we so desired, but:
* In the absence of any direct response, the status-quo provides a work-around: Omit the sub-sub-module structure and use the file-access constraints of `private`
* This overall problem probably should be solved by addressing larger questions in the Access Control scheme of Swift, irrespective of the sub-module mechanism

### Expressiveness of Sub-module Imports

One might ask: "Why can't I import only a specific sub-module or alias a sub-module?"

I have ignored this aspect of submodules because the question of `import` expressiveness is a separate issue in my mind. The fact that we cannot say:

import MyModule as Foo

Has no relationship to the lack of sub-modules in Swift.

If the community deems it an important enough use-case to warrant altering import behavior, so be it, but that can be treated as purely additive to this proposal.

But it should be understood that this approach to sub-modules is not designed to provide an expressive "exports" capability. It is primarily interested in organizing code *within* a Module

I think it's good that the proposal does not try to find solutions to these problems. We ca. More easy discuss solutions when we have been using ah modules in production for a while.

···

On 3 Mar 2017, at 16:24, Karim Nassar via swift-evolution <swift-evolution@swift.org> wrote:

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

5

Thanks for your feedback.

These are very fair concerns and point taken. There are other mechanisms we could use to achieve similar ends, I’ll work on some additional ideas.

How do you view the overall goals/use cases?

—Karim

···

On Mar 3, 2017, at 10:58 AM, Rien <Rien@Balancingrock.nl> wrote:

I don’t like the file location based approach taken, see inline.

Rien.

On 03 Mar 2017, at 16:24, Karim Nassar via swift-evolution <swift-evolution@swift.org> wrote:

I’ve read through the last couple of Swift (sub)Module proposals put forward, and since my particular use-cases for a sub-module solution seemed to be under-served by them, I’ve decided to write up my thoughts on the matter to prompt discussion.

Perhaps my use-cases are outliers, and my approach will be deemed naive by the community… I’m happy to learn better ways of doing things in Swift, and welcome any thoughts, criticism, or illumination related to these ideas.

I’m including the write-up below, but it’s also available as a gist: Yet Another Take on Swift Sub-modules · GitHub

—

# Sub-modules

A sub-module solution in Swift should have the following properties:

* Extremely light-weight
* Low API surface area
* Adopt progressive disclosure
* Integrate with Access Control features to enable a level of encapsulation & hiding between the Module and File level
* Be permeable when desired

## Discussion

As we get deeper into building real applications & frameworks with Swift, we begin to realize that having a way to express relationships between types is desireable. Currently, Swift only allows us to express these relationships at two levels, the Module and the File.

The Module boundary is acceptable for small, focused frameworks, while the File boundary is acceptable for small, focused Types, but both levels can be unweildy when dealing with certain cases where a cluster of internally related types needs to know about each other but may only want to publish a narrow set of APIs to the surrounding code, or in large complex applications which are necessarily structured as a single Module. In these cases, we wind up with large monolithic Modules or (even worse) large monolithic Files.

I have seen this proliferation of Huge Sprawling Files (HSFs) in my own code, and seek a way to combat this rising tide.

## Goals

It is a goal of this proposal to:

* Suggest a mechanism for organizing code between the Module and File levels that is as lightweight and low-friction as possible
* Provide mechanisms for authors to create both "hard" and "soft" API boundaries between the Module and File levels of their code

## Anti-Goals

It is not a goal of this proposal to:

* Move Swift away from filesystem-based organization
* Significantly alter the current Access Control philosophy of Swift

## Proposal Notes

Please take the following proposal wholely as a Straw-Man... I would be equally satisfied with any solution which meets the critera described at the top of this document.

Unless specified otherwise, all spellings proposed below are to be considered straw-men, and merely illustrative of the concepts.

## Proposed Solution

Two things are clear to me after using Swift and following the Swift Evolution list since their respective publications:

1. Swift has a preference for file-based organization
2. Vocal Swift Users dislike `fileprivate` and want to revert to Swift2-style `private`

Because of #1, this proposal does not seek to change Swift's inherent file-system organization, and instead will expand on it.

Since I personally fall into the camp described by #2, and most of the community response to this has been "Lets wait to deal with that until sub-modules", I'm making this proposal assuming that solving that quagmire is in-scope for this propsoal.

### Changes to Access Control Modifiers

As part of this proposal, I suggest the following changes to Swift 3's Access Control modifiers:

* Revert `private` to Swift 2's meaning: "hidden outside the file"
* Remove `fileprivate` as redundant

This is potentially a source-breaking change. However, it is interesting to note that this change is **not** required for the following proposal to function.

Changes that *are* necessary are:

* Change the spelling of `internal` to `module` (making `module` the new default)
* Introduce a new modifier `internal` to mean "Internal to the current sub-module and its child-sub-modules"

These changes are *not* source-breaking because the new `internal` modifier acts exactly as the old `internal` modifier unless it is used within a sub-module. The specific spelling of this new `internal` modifier is necessary to maintain backwards source compatibility.

The new `module` modifier allows authors to make APIs permeable between sub-modules while still hidden outside the owning Module if desired.

All other Access Control modifiers behave the same as they currently do irrespective of sub-module boundaries, so:

* `public` => Visible outside the Module
* `open` => Sub-classable outside the Module

### Making a Sub-module

To create a sub-module within a Module (or sub-module) is simple: The author creates a directory, and places a "sub-module declaration file" within the directory:

I don’t like “behind the scenes” magic. Any file should stand on its own. i.e. I want to see in the file itself what it can do, and what it uses from other files. Everything should be defined in the file itself.

Its place in the file system should be completely irrelevant.

//  __submodule.swift
//  MyModule

submodule SubA

Then any files within that directory are part of the sub-module:

//  Foo.swift
//  MyModule.SubA

struct Foo {
  private var mine: Bool
  internal var sub: Bool
  module var mod: Bool
}

public struct Bar {
  module var mod: Bool
  public var pub: Bool
}

This creates a sub-module called "SubA" within the module "MyModule". All files within the directory in which this file appears are understood to be contained by this sub-module.

If in the future we choose to add additional complexity (versioning, availability, etc) to the sub-module syntax, the sub-module declaration gives a natural home for this configuration.

It's important to note some benefits of this approach:

* Using the "special file" means that not all Directories are automatically submodules
* Any given source file may only be a member of 1 submodule at a time
* Use of filesystem structure to denote sub-modules plays nicely with source control
* The sub-module structure is instantly clear whether using an IDE (which can be taught to parse the `__submodule.swift` files to decorate the UI), or simple text-editor (assuming a convention of naming the Directory the same as the sub-module, which is a linter problem)

### Using Sub-modules

Referencing a sub-module should be natural and clear at this point:

#### From Within the Parent Module/Sub-module

Sub-modules are simply code-organization & namespacing tools within modules. As such, when referenced from within their parent Module, there is no need for `import`s

When I read code, the location or source of each item should be clearly traceable. Either by lexical structure or by (path) file name. If I can access the contents of another file based on the location of that other file in the file system… nope that will generate problems due to “missing file” etc.

Besides I think this could easily become a nightmare for autocomplete in IDEs ?

//  in MyModule

let foo = SubA.Foo()
foo.mine = true // Compiler error because it's private
foo.sub = true  // Compiler error because it's internal to the sub-module
foo.mod = true  // OK

#### From Outside the Parent Module/Sub-module

When referenced from outside their parent Module, one imports the whole module in the standard way:

import MyModule

let foo = SubA.Foo() // Compiler error because it's internal to the Module

let bar = SubA.Bar() // OK
bar.mod = true  // Compiler error because it's internal to the Module
bar.pub = true  // OK

## What this Proposal Deliberately Omits

This proposal deliberately omits several concepts which may be integral to various use-cases for sub-modules, primarily because they can be treated as purely additive concepts and I don't wish to weigh down the consideration of the overall approach with a larger API surface area that might be debated separately. I.e: Keep it as small as possible for now, then if it's any good, iterate on the design.

### Inter-Sub-Module Access Control

One might ask given a sub-module structure like:

MyModule
>
+--- SubA
      >
      +--- SubB

"How can SubB hide properties from MyModule without hiding them from SubA?"

This is a valid question, and not answered by this proposal for two reasons:

* This trivial case could be solved by simply adding a new modifier `submodule` if we so desired, but:
* In the absence of any direct response, the status-quo provides a work-around: Omit the sub-sub-module structure and use the file-access constraints of `private`
* This overall problem probably should be solved by addressing larger questions in the Access Control scheme of Swift, irrespective of the sub-module mechanism

### Expressiveness of Sub-module Imports

One might ask: "Why can't I import only a specific sub-module or alias a sub-module?"

I have ignored this aspect of submodules because the question of `import` expressiveness is a separate issue in my mind. The fact that we cannot say:

import MyModule as Foo

Has no relationship to the lack of sub-modules in Swift.

If the community deems it an important enough use-case to warrant altering import behavior, so be it, but that can be treated as purely additive to this proposal.

But it should be understood that this approach to sub-modules is not designed to provide an expressive "exports" capability. It is primarily interested in organizing code *within* a Module

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

6

I strongly agree with all your points. The other submodule proposals are too complex for me so I'm very happy to see a different solution which more closely fits my design priorities.

Thanks David,

Comments inline:

I’ve read through the last couple of Swift (sub)Module proposals put forward, and since my particular use-cases for a sub-module solution seemed to be under-served by them, I’ve decided to write up my thoughts on the matter to prompt discussion.

Perhaps my use-cases are outliers, and my approach will be deemed naive by the community… I’m happy to learn better ways of doing things in Swift, and welcome any thoughts, criticism, or illumination related to these ideas.

I don't think your use cases are outliers. I think your solution really feels more Swifty by starting simple.

I’m including the write-up below, but it’s also available as a gist: Yet Another Take on Swift Sub-modules · GitHub

—

# Sub-modules

A sub-module solution in Swift should have the following properties:

* Extremely light-weight
* Low API surface area
* Adopt progressive disclosure
* Integrate with Access Control features to enable a level of encapsulation & hiding between the Module and File level
* Be permeable when desired

Yes, yes, yes. Very good goals :)

:)

## Discussion

As we get deeper into building real applications & frameworks with Swift, we begin to realize that having a way to express relationships between types is desireable. Currently, Swift only allows us to express these relationships at two levels, the Module and the File.

The Module boundary is acceptable for small, focused frameworks, while the File boundary is acceptable for small, focused Types, but both levels can be unweildy when dealing with certain cases where a cluster of internally related types needs to know about each other but may only want to publish a narrow set of APIs to the surrounding code, or in large complex applications which are necessarily structured as a single Module. In these cases, we wind up with large monolithic Modules or (even worse) large monolithic Files.

I have seen this proliferation of Huge Sprawling Files (HSFs) in my own code, and seek a way to combat this rising tide.

## Goals

It is a goal of this proposal to:

* Suggest a mechanism for organizing code between the Module and File levels that is as lightweight and low-friction as possible
* Provide mechanisms for authors to create both "hard" and "soft" API boundaries between the Module and File levels of their code

## Anti-Goals

It is not a goal of this proposal to:

* Move Swift away from filesystem-based organization
* Significantly alter the current Access Control philosophy of Swift

## Proposal Notes

Please take the following proposal wholely as a Straw-Man... I would be equally satisfied with any solution which meets the critera described at the top of this document.

Unless specified otherwise, all spellings proposed below are to be considered straw-men, and merely illustrative of the concepts.

## Proposed Solution

Two things are clear to me after using Swift and following the Swift Evolution list since their respective publications:

1. Swift has a preference for file-based organization
2. Vocal Swift Users dislike `fileprivate` and want to revert to Swift2-style `private`

Because of #1, this proposal does not seek to change Swift's inherent file-system organization, and instead will expand on it.

Since I personally fall into the camp described by #2, and most of the community response to this has been "Lets wait to deal with that until sub-modules", I'm making this proposal assuming that solving that quagmire is in-scope for this propsoal.

I have posted a proposal which proposes #2. It should hopefully get merged and reviewed soon.

### Changes to Access Control Modifiers

As part of this proposal, I suggest the following changes to Swift 3's Access Control modifiers:

* Revert `private` to Swift 2's meaning: "hidden outside the file"
* Remove `fileprivate` as redundant

This is potentially a source-breaking change. However, it is interesting to note that this change is **not** required for the following proposal to function.

Changes that *are* necessary are:

* Change the spelling of `internal` to `module` (making `module` the new default)
* Introduce a new modifier `internal` to mean "Internal to the current sub-module and its child-sub-modules"

I like that you introduce only one access level and maintain source compatibility. This is also what makes the proposal fit the "progressive disclosure" goal: you don't need to learn or worry about the `module` access level until you start working with submodules.

These changes are *not* source-breaking because the new `internal` modifier acts exactly as the old `internal` modifier unless it is used within a sub-module. The specific spelling of this new `internal` modifier is necessary to maintain backwards source compatibility.

The new `module` modifier allows authors to make APIs permeable between sub-modules while still hidden outside the owning Module if desired.

All other Access Control modifiers behave the same as they currently do irrespective of sub-module boundaries, so:

* `public` => Visible outside the Module
* `open` => Sub-classable outside the Module

### Making a Sub-module

To create a sub-module within a Module (or sub-module) is simple: The author creates a directory, and places a "sub-module declaration file" within the directory:

//  __submodule.swift
//  MyModule

submodule SubA

Like the previous response, I think this potentially the weakest part of he proposal. I like the idea of a file-system based submodule system but I'm not sure I like this version of it. People are going to be very vocal about this too. Perhaps you could rewrite the proposal with several solutions for this part? Let people discuss the different solutions.

Thanks, this is exactly why I posted this original concept—to spur conversation.

An alternate approach might be to eliminate the “magic file” and directory dependency and replace them with explicit sub-module adoption per file. Something like this:

**all spellings are straw-men!**

// declared in any file

// this is the same sub-module declaration previously declared in the magic file, 
// and serves the same purpose of declaring the sub-module, 
// and acting as a home for future configuration:

submodule SubA 

// announced once in any file you wish to include in the sub-module:

in_submodule SubA

Some issues I see with this:
increases the surface area— need to introduce a new declaration for announcing sub-module inclusion
Either:
introduces a new *kind* of declaration… `in_submodule` which can only appear once in a file (which would need to be compiler-enforced), OR:
increases complexity by allowing multiple sub-module declarations in a file

I’ll think further on (and am fully open to ideas for) other approaches.

Thanks
—Karim

···

On Mar 3, 2017, at 12:35 PM, David Hart <david@hartbit.com> wrote:

On 3 Mar 2017, at 16:24, Karim Nassar via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

7

Yet another possible approach is to treat the sub-module as a symbolically-named scope (in plain-English, I’d call it a meta-type, but that already means something). This is similar mechanically to one of the earlier proposals:

submodule SubA { }

extension SubA {	
	struct Foo {  }

	submodule SubB { 
		struct Bar {  }
	}
}

extension SubA.SubB {
	struct Bast {  }
}

Pros:

* Extends an existing pattern in a somewhat reasonable-seeming way, easily discoverable

Cons:

* Though the pattern *seems* reasonable, it’s actually an odd-duck; a sub-module does not behave the same as a type in many important ways.
* There’s some impact here on how top-level declarations are evaluated (i.e.: Protocols, global statements, etc).
* Visually uglier (indentation)
* The mental model required for this is **much** more complicated, as submodule declarations can now be mixed within a file, and the (file)private Access Controls will span sub-modules.

—Karim

···

On Mar 3, 2017, at 1:13 PM, Karim Nassar <karim@karimnassar.com> wrote:

On Mar 3, 2017, at 12:35 PM, David Hart <david@hartbit.com <mailto:david@hartbit.com>> wrote:

### Making a Sub-module

To create a sub-module within a Module (or sub-module) is simple: The author creates a directory, and places a "sub-module declaration file" within the directory:

//  __submodule.swift
//  MyModule

submodule SubA

Like the previous response, I think this potentially the weakest part of he proposal. I like the idea of a file-system based submodule system but I'm not sure I like this version of it. People are going to be very vocal about this too. Perhaps you could rewrite the proposal with several solutions for this part? Let people discuss the different solutions.

Thanks, this is exactly why I posted this original concept—to spur conversation.

An alternate approach might be to eliminate the “magic file” and directory dependency and replace them with explicit sub-module adoption per file. Something like this:

**all spellings are straw-men!**

// declared in any file

// this is the same sub-module declaration previously declared in the magic file, 
// and serves the same purpose of declaring the sub-module, 
// and acting as a home for future configuration:

submodule SubA 

// announced once in any file you wish to include in the sub-module:

in_submodule SubA

Some issues I see with this:
increases the surface area— need to introduce a new declaration for announcing sub-module inclusion
Either:
introduces a new *kind* of declaration… `in_submodule` which can only appear once in a file (which would need to be compiler-enforced), OR:
increases complexity by allowing multiple sub-module declarations in a file

8

Thanks for your feedback.

These are very fair concerns and point taken. There are other mechanisms we could use to achieve similar ends, I’ll work on some additional ideas.

How do you view the overall goals/use cases?

I’m rather neutral on the need for modules/submodules. The points you mentioned don’t sound bad, but then again they are kind-a subjective.

Regards,
Rien.

···

On 03 Mar 2017, at 18:54, Karim Nassar <karim@karimnassar.com> wrote:

—Karim

On Mar 3, 2017, at 10:58 AM, Rien <Rien@Balancingrock.nl> wrote:

I don’t like the file location based approach taken, see inline.

Rien.

On 03 Mar 2017, at 16:24, Karim Nassar via swift-evolution <swift-evolution@swift.org> wrote:

I’ve read through the last couple of Swift (sub)Module proposals put forward, and since my particular use-cases for a sub-module solution seemed to be under-served by them, I’ve decided to write up my thoughts on the matter to prompt discussion.

Perhaps my use-cases are outliers, and my approach will be deemed naive by the community… I’m happy to learn better ways of doing things in Swift, and welcome any thoughts, criticism, or illumination related to these ideas.

I’m including the write-up below, but it’s also available as a gist: Yet Another Take on Swift Sub-modules · GitHub

—

# Sub-modules

A sub-module solution in Swift should have the following properties:

* Extremely light-weight
* Low API surface area
* Adopt progressive disclosure
* Integrate with Access Control features to enable a level of encapsulation & hiding between the Module and File level
* Be permeable when desired

## Discussion

As we get deeper into building real applications & frameworks with Swift, we begin to realize that having a way to express relationships between types is desireable. Currently, Swift only allows us to express these relationships at two levels, the Module and the File.

The Module boundary is acceptable for small, focused frameworks, while the File boundary is acceptable for small, focused Types, but both levels can be unweildy when dealing with certain cases where a cluster of internally related types needs to know about each other but may only want to publish a narrow set of APIs to the surrounding code, or in large complex applications which are necessarily structured as a single Module. In these cases, we wind up with large monolithic Modules or (even worse) large monolithic Files.

I have seen this proliferation of Huge Sprawling Files (HSFs) in my own code, and seek a way to combat this rising tide.

## Goals

It is a goal of this proposal to:

* Suggest a mechanism for organizing code between the Module and File levels that is as lightweight and low-friction as possible
* Provide mechanisms for authors to create both "hard" and "soft" API boundaries between the Module and File levels of their code

## Anti-Goals

It is not a goal of this proposal to:

* Move Swift away from filesystem-based organization
* Significantly alter the current Access Control philosophy of Swift

## Proposal Notes

Please take the following proposal wholely as a Straw-Man... I would be equally satisfied with any solution which meets the critera described at the top of this document.

Unless specified otherwise, all spellings proposed below are to be considered straw-men, and merely illustrative of the concepts.

## Proposed Solution

Two things are clear to me after using Swift and following the Swift Evolution list since their respective publications:

1. Swift has a preference for file-based organization
2. Vocal Swift Users dislike `fileprivate` and want to revert to Swift2-style `private`

Because of #1, this proposal does not seek to change Swift's inherent file-system organization, and instead will expand on it.

Since I personally fall into the camp described by #2, and most of the community response to this has been "Lets wait to deal with that until sub-modules", I'm making this proposal assuming that solving that quagmire is in-scope for this propsoal.

### Changes to Access Control Modifiers

As part of this proposal, I suggest the following changes to Swift 3's Access Control modifiers:

* Revert `private` to Swift 2's meaning: "hidden outside the file"
* Remove `fileprivate` as redundant

This is potentially a source-breaking change. However, it is interesting to note that this change is **not** required for the following proposal to function.

Changes that *are* necessary are:

* Change the spelling of `internal` to `module` (making `module` the new default)
* Introduce a new modifier `internal` to mean "Internal to the current sub-module and its child-sub-modules"

These changes are *not* source-breaking because the new `internal` modifier acts exactly as the old `internal` modifier unless it is used within a sub-module. The specific spelling of this new `internal` modifier is necessary to maintain backwards source compatibility.

The new `module` modifier allows authors to make APIs permeable between sub-modules while still hidden outside the owning Module if desired.

All other Access Control modifiers behave the same as they currently do irrespective of sub-module boundaries, so:

* `public` => Visible outside the Module
* `open` => Sub-classable outside the Module

### Making a Sub-module

To create a sub-module within a Module (or sub-module) is simple: The author creates a directory, and places a "sub-module declaration file" within the directory:

I don’t like “behind the scenes” magic. Any file should stand on its own. i.e. I want to see in the file itself what it can do, and what it uses from other files. Everything should be defined in the file itself.

Its place in the file system should be completely irrelevant.

//  __submodule.swift
//  MyModule

submodule SubA

Then any files within that directory are part of the sub-module:

//  Foo.swift
//  MyModule.SubA

struct Foo {
  private var mine: Bool
  internal var sub: Bool
  module var mod: Bool
}

public struct Bar {
  module var mod: Bool
  public var pub: Bool
}

This creates a sub-module called "SubA" within the module "MyModule". All files within the directory in which this file appears are understood to be contained by this sub-module.

If in the future we choose to add additional complexity (versioning, availability, etc) to the sub-module syntax, the sub-module declaration gives a natural home for this configuration.

It's important to note some benefits of this approach:

* Using the "special file" means that not all Directories are automatically submodules
* Any given source file may only be a member of 1 submodule at a time
* Use of filesystem structure to denote sub-modules plays nicely with source control
* The sub-module structure is instantly clear whether using an IDE (which can be taught to parse the `__submodule.swift` files to decorate the UI), or simple text-editor (assuming a convention of naming the Directory the same as the sub-module, which is a linter problem)

### Using Sub-modules

Referencing a sub-module should be natural and clear at this point:

#### From Within the Parent Module/Sub-module

Sub-modules are simply code-organization & namespacing tools within modules. As such, when referenced from within their parent Module, there is no need for `import`s

When I read code, the location or source of each item should be clearly traceable. Either by lexical structure or by (path) file name. If I can access the contents of another file based on the location of that other file in the file system… nope that will generate problems due to “missing file” etc.

Besides I think this could easily become a nightmare for autocomplete in IDEs ?

//  in MyModule

let foo = SubA.Foo()
foo.mine = true // Compiler error because it's private
foo.sub = true  // Compiler error because it's internal to the sub-module
foo.mod = true  // OK

#### From Outside the Parent Module/Sub-module

When referenced from outside their parent Module, one imports the whole module in the standard way:

import MyModule

let foo = SubA.Foo() // Compiler error because it's internal to the Module

let bar = SubA.Bar() // OK
bar.mod = true  // Compiler error because it's internal to the Module
bar.pub = true  // OK

## What this Proposal Deliberately Omits

This proposal deliberately omits several concepts which may be integral to various use-cases for sub-modules, primarily because they can be treated as purely additive concepts and I don't wish to weigh down the consideration of the overall approach with a larger API surface area that might be debated separately. I.e: Keep it as small as possible for now, then if it's any good, iterate on the design.

### Inter-Sub-Module Access Control

One might ask given a sub-module structure like:

MyModule
>
+--- SubA
      >
      +--- SubB

"How can SubB hide properties from MyModule without hiding them from SubA?"

This is a valid question, and not answered by this proposal for two reasons:

* This trivial case could be solved by simply adding a new modifier `submodule` if we so desired, but:
* In the absence of any direct response, the status-quo provides a work-around: Omit the sub-sub-module structure and use the file-access constraints of `private`
* This overall problem probably should be solved by addressing larger questions in the Access Control scheme of Swift, irrespective of the sub-module mechanism

### Expressiveness of Sub-module Imports

One might ask: "Why can't I import only a specific sub-module or alias a sub-module?"

I have ignored this aspect of submodules because the question of `import` expressiveness is a separate issue in my mind. The fact that we cannot say:

import MyModule as Foo

Has no relationship to the lack of sub-modules in Swift.

If the community deems it an important enough use-case to warrant altering import behavior, so be it, but that can be treated as purely additive to this proposal.

But it should be understood that this approach to sub-modules is not designed to provide an expressive "exports" capability. It is primarily interested in organizing code *within* a Module

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

9

Changes that *are* necessary are:

* Change the spelling of `internal` to `module` (making `module` the new default)
* Introduce a new modifier `internal` to mean "Internal to the current sub-module and its child-sub-modules”

Can you give concrete examples of use cases where a descendent submodule needs access to symbols declared by an ancestor? I gave some thought to this when drafting my proposal and came to the conclusion that this runs against the grain of layering and is likely to be a bad idea in practice. If there are use cases I didn’t consider I am very interested in learning about them.

On further reflection and examination of my notes, I think you’re right, and that the `internal` encapsulation should be purely horizontal. Will adjust to reflect that.

### Making a Sub-module

To create a sub-module within a Module (or sub-module) is simple: The author creates a directory, and places a "sub-module declaration file" within the directory:

```
// __submodule.swift

Why the double underscore prefix? To make it sort to the top in a file browser?

Is this file allowed to have any Swift code? Or is it limited to submodule-related declarations only? If the latter, why not use an extension such as `.submodule` or `.swiftmodule` to differentiate it from ordinary Swift files and allow the submodule to be named by the name of this file?

So, my reasoning was that by requiring a specific standard name for the declaration file, we guarantee that any directory can only describe one submodule. Prefixing the proposed name with underscores was simply a way of preventing naming collision with potential “real code” files (and yes, as a side-effect alpha-floating it to the top). Since the `submodule` declaration might expand to include statements & configuration about the sub-module, I see no reason to prohibit Swift code from existing in that sub-module declaration file… Disallowing/controlling that seems to be a style/linter concern.

However, as I mentioned above, all the specific spellings (except `internal`) for the different concepts in this proposal are straw-men awaiting input. I’d say the addition of a new type of file extension raises some concerns for me, but there’s already been a lot of push back on the general idea of using filesystem structures to organize sub-modules, so the whole idea may be moot.

Shortly, I’m going to update the original proposal gist reflecting all the comments I’ve received and part of that will be expanding the proposal section to include multiple alternate strategies for discussion, along with pros/cons for each. I’ll include a discussion on naming options for this file in this section.

If we’re going to use the file system to organize submodules this seems like a reasonable approach. It allows larger submodules to have folder hierarchies within them and also creates a central location for submodule-related declarations.

A primary flaw I see in this approach is that Xcode is the dominant IDE for Swift and the way Xcode handles files is not conducive to file-system organization. I really detest the way Xcode handles this and would vastly prefer that it simply reflected the physical file system hierarchy but I don’t think that will change any time soon. On the other hand maybe a file system based submodule system in Swift would motivate the Xcode team to better reflect the physical file system organization.

My thoughts (hopes?) on this were that should Swift adopt such a mechanism for sub-modules, then part of Xcode’s integration of the feature would be to make it aware of sub-modules in a first-class way, so that the author wouldn’t have to manually manage the project files to take advantage of the feature.

I’d envisioned either a new “Makes Sub-Module” property on Groups or a new “Sub-Module" mechanism alongside Groups & Folders in the project list. This may be wishful thinking on my part.

—Karim

···

On Mar 3, 2017, at 5:21 PM, Matthew Johnson <matthew@anandabits.com> wrote:

On Mar 3, 2017, at 9:24 AM, Karim Nassar via swift-evolution <swift-evolution@swift.org> wrote:

10

I quite love how Xcode allow me to group my source files as I want without having to move them around.
This is specially useful when working with languages that support relative #include "". Moving an header in an other group don’t break the code.

Moreover, if you want to create a folder for each group in Xcode, you can perfectly do that too as Xcode allows you to define a different path for each group. So I don’t see how it would prevent using submodule.

···

Le 3 mars 2017 à 23:21, Matthew Johnson via swift-evolution <swift-evolution@swift.org> a écrit :

On Mar 3, 2017, at 9:24 AM, Karim Nassar via swift-evolution <swift-evolution@swift.org> wrote:

I’ve read through the last couple of Swift (sub)Module proposals put forward, and since my particular use-cases for a sub-module solution seemed to be under-served by them, I’ve decided to write up my thoughts on the matter to prompt discussion.

Perhaps my use-cases are outliers, and my approach will be deemed naive by the community… I’m happy to learn better ways of doing things in Swift, and welcome any thoughts, criticism, or illumination related to these ideas.

I’m including the write-up below, but it’s also available as a gist: Yet Another Take on Swift Sub-modules · GitHub

—

# Sub-modules

A sub-module solution in Swift should have the following properties:

* Extremely light-weight
* Low API surface area
* Adopt progressive disclosure
* Integrate with Access Control features to enable a level of encapsulation & hiding between the Module and File level
* Be permeable when desired

## Discussion

As we get deeper into building real applications & frameworks with Swift, we begin to realize that having a way to express relationships between types is desireable. Currently, Swift only allows us to express these relationships at two levels, the Module and the File.

The Module boundary is acceptable for small, focused frameworks, while the File boundary is acceptable for small, focused Types, but both levels can be unweildy when dealing with certain cases where a cluster of internally related types needs to know about each other but may only want to publish a narrow set of APIs to the surrounding code, or in large complex applications which are necessarily structured as a single Module. In these cases, we wind up with large monolithic Modules or (even worse) large monolithic Files.

I have seen this proliferation of Huge Sprawling Files (HSFs) in my own code, and seek a way to combat this rising tide.

## Goals

It is a goal of this proposal to:

* Suggest a mechanism for organizing code between the Module and File levels that is as lightweight and low-friction as possible
* Provide mechanisms for authors to create both "hard" and "soft" API boundaries between the Module and File levels of their code

## Anti-Goals

It is not a goal of this proposal to:

* Move Swift away from filesystem-based organization
* Significantly alter the current Access Control philosophy of Swift

## Proposal Notes

Please take the following proposal wholely as a Straw-Man... I would be equally satisfied with any solution which meets the critera described at the top of this document.

Unless specified otherwise, all spellings proposed below are to be considered straw-men, and merely illustrative of the concepts.

## Proposed Solution

Two things are clear to me after using Swift and following the Swift Evolution list since their respective publications:

1. Swift has a preference for file-based organization
2. Vocal Swift Users dislike `fileprivate` and want to revert to Swift2-style `private`

Because of #1, this proposal does not seek to change Swift's inherent file-system organization, and instead will expand on it.

Since I personally fall into the camp described by #2, and most of the community response to this has been "Lets wait to deal with that until sub-modules", I'm making this proposal assuming that solving that quagmire is in-scope for this propsoal.

### Changes to Access Control Modifiers

As part of this proposal, I suggest the following changes to Swift 3's Access Control modifiers:

* Revert `private` to Swift 2's meaning: "hidden outside the file"
* Remove `fileprivate` as redundant

This is potentially a source-breaking change. However, it is interesting to note that this change is **not** required for the following proposal to function.

Changes that *are* necessary are:

* Change the spelling of `internal` to `module` (making `module` the new default)
* Introduce a new modifier `internal` to mean "Internal to the current sub-module and its child-sub-modules”

Can you give concrete examples of use cases where a descendent submodule needs access to symbols declared by an ancestor? I gave some thought to this when drafting my proposal and came to the conclusion that this runs against the grain of layering and is likely to be a bad idea in practice. If there are use cases I didn’t consider I am very interested in learning about them.

These changes are *not* source-breaking because the new `internal` modifier acts exactly as the old `internal` modifier unless it is used within a sub-module. The specific spelling of this new `internal` modifier is necessary to maintain backwards source compatibility.

The new `module` modifier allows authors to make APIs permeable between sub-modules while still hidden outside the owning Module if desired.

All other Access Control modifiers behave the same as they currently do irrespective of sub-module boundaries, so:

* `public` => Visible outside the Module
* `open` => Sub-classable outside the Module

### Making a Sub-module

To create a sub-module within a Module (or sub-module) is simple: The author creates a directory, and places a "sub-module declaration file" within the directory:

//  __submodule.swift

Why the double underscore prefix? To make it sort to the top in a file browser?

Is this file allowed to have any Swift code? Or is it limited to submodule-related declarations only? If the latter, why not use an extension such as .submodule or .swiftmodule to differentiate it from ordinary Swift files and allow the submodule to be named by the name of this file?

// MyModule

submodule SubA


Then any files within that directory are part of the sub\-module:

// Foo.swift
// MyModule.SubA

struct Foo {
private var mine: Bool
internal var sub: Bool
module var mod: Bool
}

public struct Bar {
module var mod: Bool
public var pub: Bool
}


This creates a sub\-module called &quot;SubA&quot; within the module &quot;MyModule&quot;\. All files within the directory in which this file appears are understood to be contained by this sub\-module\.

If in the future we choose to add additional complexity \(versioning, \#availability, etc\) to the sub\-module syntax, the sub\-module declaration gives a natural home for this configuration\.

It&#39;s important to note some benefits of this approach:

\* Using the &quot;special file&quot; means that not all Directories are automatically submodules
\* Any given source file may only be a member of 1 submodule at a time
\* Use of filesystem structure to denote sub\-modules plays nicely with source control
\* The sub\-module structure is instantly clear whether using an IDE \(which can be taught to parse the \`\_\_submodule\.swift\` files to decorate the UI\), or simple text\-editor \(assuming a convention of naming the Directory the same as the sub\-module, which is a linter problem\)

If we’re going to use the file system to organize submodules this seems like a reasonable approach. It allows larger submodules to have folder hierarchies within them and also creates a central location for submodule-related declarations.

A primary flaw I see in this approach is that Xcode is the dominant IDE for Swift and the way Xcode handles files is not conducive to file-system organization. I really detest the way Xcode handles this and would vastly prefer that it simply reflected the physical file system hierarchy but I don’t think that will change any time soon. On the other hand maybe a file system based submodule system in Swift would motivate the Xcode team to better reflect the physical file system organization.

11

I don’t mind Xcode having this feature. What I don’t like is that Xcode makes it way more difficult than it should be to keep groups in sync with the file system if that is what you want.

The problem I see is that if a submodule design follows the physical structure of the file system that is the view you should have in the IDE as well. If it doesn’t you have no indication at all of what submodule a file is in (unless it offers some other kind of indication of that). I think this is worth considering as we discuss the design of submodules in Swift.

···

On Mar 4, 2017, at 8:12 AM, Jean-Daniel <mailing@xenonium.com> wrote:

Le 3 mars 2017 à 23:21, Matthew Johnson via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> a écrit :

On Mar 3, 2017, at 9:24 AM, Karim Nassar via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

I’ve read through the last couple of Swift (sub)Module proposals put forward, and since my particular use-cases for a sub-module solution seemed to be under-served by them, I’ve decided to write up my thoughts on the matter to prompt discussion.

Perhaps my use-cases are outliers, and my approach will be deemed naive by the community… I’m happy to learn better ways of doing things in Swift, and welcome any thoughts, criticism, or illumination related to these ideas.

I’m including the write-up below, but it’s also available as a gist: Yet Another Take on Swift Sub-modules · GitHub

—

# Sub-modules

A sub-module solution in Swift should have the following properties:

* Extremely light-weight
* Low API surface area
* Adopt progressive disclosure
* Integrate with Access Control features to enable a level of encapsulation & hiding between the Module and File level
* Be permeable when desired

## Discussion

As we get deeper into building real applications & frameworks with Swift, we begin to realize that having a way to express relationships between types is desireable. Currently, Swift only allows us to express these relationships at two levels, the Module and the File.

The Module boundary is acceptable for small, focused frameworks, while the File boundary is acceptable for small, focused Types, but both levels can be unweildy when dealing with certain cases where a cluster of internally related types needs to know about each other but may only want to publish a narrow set of APIs to the surrounding code, or in large complex applications which are necessarily structured as a single Module. In these cases, we wind up with large monolithic Modules or (even worse) large monolithic Files.

I have seen this proliferation of Huge Sprawling Files (HSFs) in my own code, and seek a way to combat this rising tide.

## Goals

It is a goal of this proposal to:

* Suggest a mechanism for organizing code between the Module and File levels that is as lightweight and low-friction as possible
* Provide mechanisms for authors to create both "hard" and "soft" API boundaries between the Module and File levels of their code

## Anti-Goals

It is not a goal of this proposal to:

* Move Swift away from filesystem-based organization
* Significantly alter the current Access Control philosophy of Swift

## Proposal Notes

Please take the following proposal wholely as a Straw-Man... I would be equally satisfied with any solution which meets the critera described at the top of this document.

Unless specified otherwise, all spellings proposed below are to be considered straw-men, and merely illustrative of the concepts.

## Proposed Solution

Two things are clear to me after using Swift and following the Swift Evolution list since their respective publications:

1. Swift has a preference for file-based organization
2. Vocal Swift Users dislike `fileprivate` and want to revert to Swift2-style `private`

Because of #1, this proposal does not seek to change Swift's inherent file-system organization, and instead will expand on it.

Since I personally fall into the camp described by #2, and most of the community response to this has been "Lets wait to deal with that until sub-modules", I'm making this proposal assuming that solving that quagmire is in-scope for this propsoal.

### Changes to Access Control Modifiers

As part of this proposal, I suggest the following changes to Swift 3's Access Control modifiers:

* Revert `private` to Swift 2's meaning: "hidden outside the file"
* Remove `fileprivate` as redundant

This is potentially a source-breaking change. However, it is interesting to note that this change is **not** required for the following proposal to function.

Changes that *are* necessary are:

* Change the spelling of `internal` to `module` (making `module` the new default)
* Introduce a new modifier `internal` to mean "Internal to the current sub-module and its child-sub-modules”

Can you give concrete examples of use cases where a descendent submodule needs access to symbols declared by an ancestor? I gave some thought to this when drafting my proposal and came to the conclusion that this runs against the grain of layering and is likely to be a bad idea in practice. If there are use cases I didn’t consider I am very interested in learning about them.

These changes are *not* source-breaking because the new `internal` modifier acts exactly as the old `internal` modifier unless it is used within a sub-module. The specific spelling of this new `internal` modifier is necessary to maintain backwards source compatibility.

The new `module` modifier allows authors to make APIs permeable between sub-modules while still hidden outside the owning Module if desired.

All other Access Control modifiers behave the same as they currently do irrespective of sub-module boundaries, so:

* `public` => Visible outside the Module
* `open` => Sub-classable outside the Module

### Making a Sub-module

To create a sub-module within a Module (or sub-module) is simple: The author creates a directory, and places a "sub-module declaration file" within the directory:

//  __submodule.swift

Why the double underscore prefix? To make it sort to the top in a file browser?

Is this file allowed to have any Swift code? Or is it limited to submodule-related declarations only? If the latter, why not use an extension such as .submodule or .swiftmodule to differentiate it from ordinary Swift files and allow the submodule to be named by the name of this file?

// MyModule

submodule SubA


Then any files within that directory are part of the sub\-module:

// Foo.swift
// MyModule.SubA

struct Foo {
private var mine: Bool
internal var sub: Bool
module var mod: Bool
}

public struct Bar {
module var mod: Bool
public var pub: Bool
}


This creates a sub\-module called &quot;SubA&quot; within the module &quot;MyModule&quot;\. All files within the directory in which this file appears are understood to be contained by this sub\-module\.

If in the future we choose to add additional complexity \(versioning, \#availability, etc\) to the sub\-module syntax, the sub\-module declaration gives a natural home for this configuration\.

It&#39;s important to note some benefits of this approach:

\* Using the &quot;special file&quot; means that not all Directories are automatically submodules
\* Any given source file may only be a member of 1 submodule at a time
\* Use of filesystem structure to denote sub\-modules plays nicely with source control
\* The sub\-module structure is instantly clear whether using an IDE \(which can be taught to parse the \`\_\_submodule\.swift\` files to decorate the UI\), or simple text\-editor \(assuming a convention of naming the Directory the same as the sub\-module, which is a linter problem\)

If we’re going to use the file system to organize submodules this seems like a reasonable approach. It allows larger submodules to have folder hierarchies within them and also creates a central location for submodule-related declarations.

A primary flaw I see in this approach is that Xcode is the dominant IDE for Swift and the way Xcode handles files is not conducive to file-system organization. I really detest the way Xcode handles this and would vastly prefer that it simply reflected the physical file system hierarchy but I don’t think that will change any time soon. On the other hand maybe a file system based submodule system in Swift would motivate the Xcode team to better reflect the physical file system organization.

I quite love how Xcode allow me to group my source files as I want without having to move them around.
This is specially useful when working with languages that support relative #include "". Moving an header in an other group don’t break the code.

Moreover, if you want to create a folder for each group in Xcode, you can perfectly do that too as Xcode allows you to define a different path for each group. So I don’t see how it would prevent using submodule.

12

Here is a new gist with updates based on all the great feedback I’ve received (accidentally created the first one anonymously):

And inline:

# Sub-modules

A sub-module solution in Swift should have the following properties:

* Extremely light-weight
* Low API surface area
* Adopt progressive disclosure
* Integrate with Access Control features to enable a level of encapsulation & hiding between the Module and File level
* Be permeable when desired

## Discussion

As we get deeper into building real applications & frameworks with Swift, we begin to realize that having a way to express relationships between types is desireable. Currently, Swift only allows us to express these relationships at two levels, the Module and the File.

The Module boundary is acceptable for small, focused frameworks, while the File boundary is acceptable for small, focused Types, but both levels can be unweildy when dealing with certain cases where a cluster of internally related types needs to know about each other but may only want to publish a narrow set of APIs to the surrounding code, or in large complex applications which are necessarily structured as a single Module. In these cases, we wind up with large monolithic Modules or (even worse) large monolithic Files.

I have seen this proliferation of Huge Sprawling Files (HSFs) in my own code, and seek a way to combat this rising tide.

## Goals

It is a goal of this proposal to:

* Suggest a mechanism for organizing code between the Module and File levels that is as lightweight and low-friction as possible
* Provide mechanisms for authors to create both "hard" and "soft" API boundaries between the Module and File levels of their code

## Anti-Goals

It is not a goal of this proposal to:

* Move Swift away from filesystem-based organization
* Significantly alter the current Access Control philosophy of Swift

## Proposal Notes

Please take the following proposal wholely as a Straw-Man... I would be equally satisfied with any solution which meets the critera described at the top of this document.

Unless specified otherwise, all spellings proposed below are to be considered straw-men, and merely illustrative of the concepts.

# Proposed Solution

Two things are clear to me after using Swift and following the Swift Evolution list since their respective publications:

1. Swift has a preference for file-based organization
2. Vocal Swift Users dislike `fileprivate` and want to revert to Swift2-style `private`

Because of #1, this proposal does not seek to change Swift's inherent file-system organization, and instead will expand on it.

Since I personally fall into the camp described by #2, and most of the community response to this has been "Lets wait to deal with that until sub-modules", I'm making this proposal assuming that solving that quagmire is in-scope for this propsoal.

## Changes to Access Control Modifiers

As part of this proposal (or as a pre-requisite assumption), I suggest the following changes to Swift 3's Access Control modifiers:

* Revert `private` to Swift 2's meaning: "hidden outside the file"
* Remove `fileprivate` as redundant

This is potentially a source-breaking change. However, it is interesting to note that this change is **not** required for the following proposal to function.

Changes that *are* necessary are:

* Change the spelling of `internal` to `module` (making `module` the new default)
* Introduce a new modifier `internal` to mean "Internal to the current sub-module (or if at the 'top' level of a module, Internal to the top-level, ie: hidden from it's sub-modules)"

These changes are *not* source-breaking because the new `internal` modifier acts exactly as the old `internal` modifier when used without sub-modules. The specific spelling of this new `internal` modifier is necessary to maintain backwards source compatibility.

The new `module` modifier allows authors to make APIs permeable between sub-modules while still hidden outside the owning Module if desired.

All other Access Control modifiers behave the same as they currently do irrespective of sub-module boundaries, so:

* `public` => Visible outside the Module
* `open` => Sub-classable outside the Module

## Making a Sub-module

There are various alternative mechanisms that might be chosen for declaring a sub-module. I describe 3 of them here, each with some observations, pros, and cons. The intent of this proposal is to select **one** mechanism that best balances the proposal goals and the mechanism's compromises. Other alternatives are welome.

### Alternative 1: Using the Filesystem

To create a sub-module within a Module (or sub-module) is simple: The author creates a directory, and places a "sub-module declaration file" within the directory:

//  __submodule.swift
//  MyModule

submodule SubA

Then any files within that directory are part of the sub-module:

//  Foo.swift
//  MyModule.SubA

struct Foo {
    private var mine: Bool
    internal var sub: Bool
    module var mod: Bool
}

public struct Bar {
    module var mod: Bool
    public var pub: Bool
}

This creates a sub-module called "SubA" within the module "MyModule". All files within the directory in which this file appears are understood to be contained by this sub-module.

Nested sub-modules are created by the same structure in nested directories

//  __submodule.swift
//  MyModule.SubA

submodule SubB


//  Bast.swift
//  MyModule.SubA.SubB

module struct Bast {
}

If in the future we choose to add additional complexity (versioning, #availability, etc) to the sub-module syntax, the sub-module declaration gives a natural home for this configuration:

submodule SubA {
    // configuration goes here 
}

An alternative to the proposed `__submodule.swift` might be to introduce a new extension `.swiftsubmodule` or somesuch. This choice may have additional benefits or drawbacks.

#### Benefits

* Any given source file may only be a member of 1 submodule at a time
* Use of filesystem structure to denote sub-modules plays nicely with source control
* The sub-module structure is instantly clear whether using an IDE (which can be taught to parse the `__submodule.swift` files to decorate the UI), or simple text-editor (assuming a convention of naming the Directory the same as the sub-module, which is a linter problem)

#### Drawbacks

* Code-meaning becomes dependent on the code file's position within the filesystem
* Requires special compiler rules to parse & validate the new declaration file
* `submodule` may only appear in a sub-module declaration file
* `submodule` may only appear once in a file

#### Observations

* Using the "special file" means that not all Directories are automatically submodules
* The naming of this file should attempt to prevent accidental collision with source files the author might choose
* If adopted, it is my hope that IDE's (Xcode) would provide first-class support for this mechanism, so that authors would not have to manually manage file-system representation of sub-module memberships

### Alternative 2: In Situ Declaration & Adoption

A sub-module is declared within the module in any source file:

submodule SubA

Files which are to be members of the sub-module declare their membership with a new declaration `in_submodule` which is only allowed to appear **once** in any source file to prevent the complexity of multi-sub-module memberships.

in_submodule SubA 

struct Foo {
    private var mine: Bool
    internal var sub: Bool
    module var mod: Bool
}

public struct Bar {
    module var mod: Bool
    public var pub: Bool
}

Nested sub-modules are declared & adopted with dot-syntax

submodule SubA.SubB 

in_submodule SubA.SubB

module struct Bast {

}

#### Benefits

* Pure in-code sub-module definition, with no dependencies on filesystem location

#### Drawbacks

* Increases the surface area— we need to introduce a new declaration for announcing sub-module inclusion
* Requires special compiler rules to parse & validate the new adoption declaration
* `in_submodule` may only appear once in a file

#### Observations

* Future iteration might choose to relax the "one sub-module membership only" rule (albeit at significant authoring complexity)

### Alternative 3: Scope-like Sub-modules

In this alternative, declaration looks similar to Alternative 2, but its semantic meaning has changed:

The sub-module declaration represents a symbolic entity, following the established rules for type declarations:

submodule SubA {

    struct Foo {  }

}

Memberships are expressed either by declaring types within the submodule declaration itself, or by leveraging the `extension` syntax:

extension SubA {	

   	public struct Bar {  }
    
    submodule SubB { 
        module struct Bast {  }
    }

}

#### Benefits

* Extends an existing pattern in a somewhat reasonable-seeming way
* easily discoverable

#### Drawbacks

* Will require changes to how top-level declarations are parsed & evaluated (i.e.: Protocols, global statements, operator funcs, etc)
* Visually uglier due to indentation

#### Observations

* Though the pattern *seems* reasonable, it’s actually an odd-duck; a sub-module does not behave the same as a type in many important ways, and the use of the Type-like syntax mechanisms may lead (esp. novice) authors to expect behaviors which cannot be met (ie: "Why can't sub-modules have properties?")
* The mental model required for this approach is **much** more complicated, as submodule declarations can now be mixed within a file, and the (file)private Access Controls will span sub-modules

## Using Sub-modules

Referencing a sub-module should be natural and clear:

### From Within the Parent Module/Sub-module

Sub-modules are simply code-organization & namespacing tools within modules. As such, when referenced from within their parent Module, there is no need for `import`s

//  in MyModule

let foo = SubA.Foo()
foo.mine = true // Compiler error because it's private
foo.sub = true  // Compiler error because it's internal to the sub-module
foo.mod = true  // OK

### From Outside the Parent Module/Sub-module

When referenced from outside their parent Module, one imports the whole module in the standard way:

import MyModule

let foo = SubA.Foo() // Compiler error because it's internal to the Module

let bar = SubA.Bar() // OK
bar.mod = true  // Compiler error because it's internal to the Module
bar.pub = true  // OK

## What this Proposal Deliberately Omits

This proposal deliberately omits several concepts which may be integral to various use-cases for sub-modules, primarily because they can be treated as purely additive concepts and I don't wish to weigh down the consideration of the overall approach with a larger API surface area that might be debated separately. I.e: Keep it as small as possible for now, then if it's any good, iterate on the design.

### Inter-Sub-Module Access Control

One might ask given a sub-module structure like:

MyModule
  >
  +--- SubA
        >
        +--- SubB

"How can SubB hide properties from MyModule without hiding them from SubA?"

This is a valid question, and not answered by this proposal for two reasons:

* This trivial case could be solved by simply adding a new modifier `submodule` if we so desired, but:
* In the absence of any direct response, the status-quo provides a work-around: Omit the sub-sub-module structure and use the file-access constraints of `private`
* This overall problem probably should be solved by addressing larger questions in the Access Control scheme of Swift, irrespective of the sub-module mechanism

### Expressiveness of Sub-module Imports

One might ask: "Why can't I import only a specific sub-module or alias a sub-module?"

I have ignored this aspect of submodules because the question of `import` expressiveness is a separate issue in my mind. The fact that we cannot say:

import MyModule as Foo

Has no relationship to the lack of sub-modules in Swift.

If the community deems it an important enough use-case to warrant altering import behavior, so be it, but that can be treated as purely additive to this proposal.

But it should be understood that this approach to sub-modules is not designed to provide an expressive "exports" capability. It is primarily interested in organizing code *within* a Module

13

Changes that *are* necessary are:

* Change the spelling of `internal` to `module` (making `module` the new default)
* Introduce a new modifier `internal` to mean "Internal to the current sub-module and its child-sub-modules”

Can you give concrete examples of use cases where a descendent submodule needs access to symbols declared by an ancestor? I gave some thought to this when drafting my proposal and came to the conclusion that this runs against the grain of layering and is likely to be a bad idea in practice. If there are use cases I didn’t consider I am very interested in learning about them.

On further reflection and examination of my notes, I think you’re right, and that the `internal` encapsulation should be purely horizontal. Will adjust to reflect that.

### Making a Sub-module

To create a sub-module within a Module (or sub-module) is simple: The author creates a directory, and places a "sub-module declaration file" within the directory:

```
// __submodule.swift

Why the double underscore prefix? To make it sort to the top in a file browser?

Is this file allowed to have any Swift code? Or is it limited to submodule-related declarations only? If the latter, why not use an extension such as `.submodule` or `.swiftmodule` to differentiate it from ordinary Swift files and allow the submodule to be named by the name of this file?

So, my reasoning was that by requiring a specific standard name for the declaration file, we guarantee that any directory can only describe one submodule. Prefixing the proposed name with underscores was simply a way of preventing naming collision with potential “real code” files (and yes, as a side-effect alpha-floating it to the top). Since the `submodule` declaration might expand to include statements & configuration about the sub-module, I see no reason to prohibit Swift code from existing in that sub-module declaration file… Disallowing/controlling that seems to be a style/linter concern.

However, as I mentioned above, all the specific spellings (except `internal`) for the different concepts in this proposal are straw-men awaiting input. I’d say the addition of a new type of file extension raises some concerns for me, but there’s already been a lot of push back on the general idea of using filesystem structures to organize sub-modules, so the whole idea may be moot.

I’ve actually been starting to come around to the idea of using the file system. Not so much because I really like it, but because I have been considering further some of the drawbacks of other approaches.

One big reason is that a submodule should form a scope and scopes should consist of code that is physically adjacent. In practice this means it should reside in close proximity in the file system. Allowing any file in a project to be a part of any submodule partly defeats the purpose of using them to structure a project internally. If we’re going to be organizing the files in a submodule physically anyway maybe we should just take advantage of that fact and prevent a stray file in a distant part of the file system from being part of the submodule.

The second reason is that there is a big problem with placing a simple `submodule Foo` declaration at the top of each file in a submodule. We all make typos from time to time. This method makes it too easy to accidentally type `submodule Fooo` and end up in a submodule you didn’t intend. This mistake would likely be caught relatively quickly but it seems silly to have a system subject to this kind of mistake. This means we would end arbitrarily placing the declaration in one file and saying `extends submodule Foo` in the rest of the files. Your design avoids the need to arbitrarily choose where to place the declaration and avoids the need for a declaration in the rest of the files.

Shortly, I’m going to update the original proposal gist reflecting all the comments I’ve received and part of that will be expanding the proposal section to include multiple alternate strategies for discussion, along with pros/cons for each. I’ll include a discussion on naming options for this file in this section.

I’m going to look at your new draft this afternoon while I’m working on updating my proposal. I think you’ll find it much more to your liking than the original draft. Perhaps we will be able to converge if we keep the discussion going.

If we’re going to use the file system to organize submodules this seems like a reasonable approach. It allows larger submodules to have folder hierarchies within them and also creates a central location for submodule-related declarations.

A primary flaw I see in this approach is that Xcode is the dominant IDE for Swift and the way Xcode handles files is not conducive to file-system organization. I really detest the way Xcode handles this and would vastly prefer that it simply reflected the physical file system hierarchy but I don’t think that will change any time soon. On the other hand maybe a file system based submodule system in Swift would motivate the Xcode team to better reflect the physical file system organization.

My thoughts (hopes?) on this were that should Swift adopt such a mechanism for sub-modules, then part of Xcode’s integration of the feature would be to make it aware of sub-modules in a first-class way, so that the author wouldn’t have to manually manage the project files to take advantage of the feature.

I’d envisioned either a new “Makes Sub-Module” property on Groups or a new “Sub-Module" mechanism alongside Groups & Folders in the project list. This may be wishful thinking on my part.

The problem with a "Makes Sub-Module” on a group in Xcode is that groups do not mirror the file system. Maybe a file system based submodule feature would motivate the Xcode team to support group structures that do mirror the file system better. I hope that would be the case! Unfortunately I don’t think we’d have a guarantee when reviewing the feature and there are significant practical implications for using the feature in Xcode without that kind of support. We would probably just have to trust Apple to get this right.

···

On Mar 4, 2017, at 9:56 AM, Karim Nassar <karim@karimnassar.com> wrote:

On Mar 3, 2017, at 5:21 PM, Matthew Johnson <matthew@anandabits.com <mailto:matthew@anandabits.com>> wrote:

On Mar 3, 2017, at 9:24 AM, Karim Nassar via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

—Karim

14

Changes that *are* necessary are:

* Change the spelling of `internal` to `module` (making `module` the new default)
* Introduce a new modifier `internal` to mean "Internal to the current sub-module and its child-sub-modules”

Can you give concrete examples of use cases where a descendent submodule needs access to symbols declared by an ancestor? I gave some thought to this when drafting my proposal and came to the conclusion that this runs against the grain of layering and is likely to be a bad idea in practice. If there are use cases I didn’t consider I am very interested in learning about them.

On further reflection and examination of my notes, I think you’re right, and that the `internal` encapsulation should be purely horizontal. Will adjust to reflect that.

### Making a Sub-module

To create a sub-module within a Module (or sub-module) is simple: The author creates a directory, and places a "sub-module declaration file" within the directory:

```
// __submodule.swift

Why the double underscore prefix? To make it sort to the top in a file browser?

Is this file allowed to have any Swift code? Or is it limited to submodule-related declarations only? If the latter, why not use an extension such as `.submodule` or `.swiftmodule` to differentiate it from ordinary Swift files and allow the submodule to be named by the name of this file?

So, my reasoning was that by requiring a specific standard name for the declaration file, we guarantee that any directory can only describe one submodule. Prefixing the proposed name with underscores was simply a way of preventing naming collision with potential “real code” files (and yes, as a side-effect alpha-floating it to the top). Since the `submodule` declaration might expand to include statements & configuration about the sub-module, I see no reason to prohibit Swift code from existing in that sub-module declaration file… Disallowing/controlling that seems to be a style/linter concern.

However, as I mentioned above, all the specific spellings (except `internal`) for the different concepts in this proposal are straw-men awaiting input. I’d say the addition of a new type of file extension raises some concerns for me, but there’s already been a lot of push back on the general idea of using filesystem structures to organize sub-modules, so the whole idea may be moot.

I’ve actually been starting to come around to the idea of using the file system. Not so much because I really like it, but because I have been considering further some of the drawbacks of other approaches.

One big reason is that a submodule should form a scope and scopes should consist of code that is physically adjacent. In practice this means it should reside in close proximity in the file system. Allowing any file in a project to be a part of any submodule partly defeats the purpose of using them to structure a project internally. If we’re going to be organizing the files in a submodule physically anyway maybe we should just take advantage of that fact and prevent a stray file in a distant part of the file system from being part of the submodule.

The second reason is that there is a big problem with placing a simple `submodule Foo` declaration at the top of each file in a submodule. We all make typos from time to time. This method makes it too easy to accidentally type `submodule Fooo` and end up in a submodule you didn’t intend. This mistake would likely be caught relatively quickly but it seems silly to have a system subject to this kind of mistake. This means we would end arbitrarily placing the declaration in one file and saying `extends submodule Foo` in the rest of the files. Your design avoids the need to arbitrarily choose where to place the declaration and avoids the need for a declaration in the rest of the files.

I’m not dead-set on this approach, but as you say, it solves a *lot* of problems that other approaches introduce. I do recognize the reasonableness of the main argument against, that file location shouldn’t have such a dramatic affect on behavior… *but* the fact is (whether by convention or expediency) we already *do* have filesystem location dependencies on our code projects…

* One can’t willy-nilly move files around the disk and expect a project to build… if you move it somewhere the compiler doesn’t know to look for it, you’re going to break things
* One can’t just move a file out of your SCM repo root, or you’ll “lose” the file

One might argue that these are pathological cases, but the fact remains that this approach only introduces a new constraint on project structure, not a new *kind* of constraint, and it is one that is purely optional. You don’t *have* to use sub-modules in a project, and if you don’t, there’s no additive burden on you.

Shortly, I’m going to update the original proposal gist reflecting all the comments I’ve received and part of that will be expanding the proposal section to include multiple alternate strategies for discussion, along with pros/cons for each. I’ll include a discussion on naming options for this file in this section.

I’m going to look at your new draft this afternoon while I’m working on updating my proposal. I think you’ll find it much more to your liking than the original draft. Perhaps we will be able to converge if we keep the discussion going.

I’d be happy to cooperate-with/converge-with/withdraw-in-favor-of. I’m mostly interested in a solid, light-weight solution to my use-cases, however that looks. I look forward to seeing your updates!

If we’re going to use the file system to organize submodules this seems like a reasonable approach. It allows larger submodules to have folder hierarchies within them and also creates a central location for submodule-related declarations.

A primary flaw I see in this approach is that Xcode is the dominant IDE for Swift and the way Xcode handles files is not conducive to file-system organization. I really detest the way Xcode handles this and would vastly prefer that it simply reflected the physical file system hierarchy but I don’t think that will change any time soon. On the other hand maybe a file system based submodule system in Swift would motivate the Xcode team to better reflect the physical file system organization.

My thoughts (hopes?) on this were that should Swift adopt such a mechanism for sub-modules, then part of Xcode’s integration of the feature would be to make it aware of sub-modules in a first-class way, so that the author wouldn’t have to manually manage the project files to take advantage of the feature.

I’d envisioned either a new “Makes Sub-Module” property on Groups or a new “Sub-Module" mechanism alongside Groups & Folders in the project list. This may be wishful thinking on my part.

The problem with a "Makes Sub-Module” on a group in Xcode is that groups do not mirror the file system. Maybe a file system based submodule feature would motivate the Xcode team to support group structures that do mirror the file system better. I hope that would be the case! Unfortunately I don’t think we’d have a guarantee when reviewing the feature and there are significant practical implications for using the feature in Xcode without that kind of support. We would probably just have to trust Apple to get this right.

Too true. I don’t know how much the Swift team can influence or contribute to Xcode. On the other hand, using Xcode’s “Folder References” feature isn’t so onerous that it would prevent me from getting very large benefit from sub-modules in my projects.

···

On Mar 4, 2017, at 2:54 PM, Matthew Johnson <matthew@anandabits.com> wrote:

On Mar 4, 2017, at 9:56 AM, Karim Nassar <karim@karimnassar.com <mailto:karim@karimnassar.com>> wrote:

On Mar 3, 2017, at 5:21 PM, Matthew Johnson <matthew@anandabits.com <mailto:matthew@anandabits.com>> wrote:

On Mar 3, 2017, at 9:24 AM, Karim Nassar via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

—

As one concrete example, I have a large app that imports a Module (Framework) which represents a kind of domain-specific DOM graph. There is a Query type which must live within that module (both for WMO, and for more esoteric dependency reasons). This Query type has 11 related or sub- types including specialized graph nodes and a parser. Out of these 12 types, the API genuinely exposes 2 types (Query and QueryError) and a grand total of 5 methods/properties outside of the “Query” cluster of types. In order to achieve this encapsulation, I’ve had to lump all of these into a single 2k+ lines-long file with heavy use of fileprivate, which is a real pain to manage in terms of reading, editing, and SCM.

One might say, oh, that’s not so bad, except this is one example of many similar situations in a very large codebase consisting of the Application and 9 subordinate Modules.

With this or similar sub-module system, this code could easily be factored into a tight cluster of specialized files, with no one file being larger than a couple of hundred lines, and most being in the 10–50 line neighborhood—much more manageable.

In fact, I’d go so far as to say that if I had a real sub-module system, many of those Modules would get demoted. The cost of maintaining full frameworks for code that will never be used outside of this application just to achieve encapsulation isn’t trivial.

And another neat thing about a directory-based approach as I’ve proposed is that if I ever do want to “promote” a sub-module to its own Module, it’s very easy to do because I know where *all* the files are, and all of the sub-module’s child sub-modules automatically get “promoted up” one level without any effort.

—Karim

15

Changes that *are* necessary are:

* Change the spelling of `internal` to `module` (making `module` the new default)
* Introduce a new modifier `internal` to mean "Internal to the current sub-module and its child-sub-modules”

Can you give concrete examples of use cases where a descendent submodule needs access to symbols declared by an ancestor? I gave some thought to this when drafting my proposal and came to the conclusion that this runs against the grain of layering and is likely to be a bad idea in practice. If there are use cases I didn’t consider I am very interested in learning about them.

On further reflection and examination of my notes, I think you’re right, and that the `internal` encapsulation should be purely horizontal. Will adjust to reflect that.

### Making a Sub-module

To create a sub-module within a Module (or sub-module) is simple: The author creates a directory, and places a "sub-module declaration file" within the directory:

```
// __submodule.swift

Why the double underscore prefix? To make it sort to the top in a file browser?

Is this file allowed to have any Swift code? Or is it limited to submodule-related declarations only? If the latter, why not use an extension such as `.submodule` or `.swiftmodule` to differentiate it from ordinary Swift files and allow the submodule to be named by the name of this file?

So, my reasoning was that by requiring a specific standard name for the declaration file, we guarantee that any directory can only describe one submodule. Prefixing the proposed name with underscores was simply a way of preventing naming collision with potential “real code” files (and yes, as a side-effect alpha-floating it to the top). Since the `submodule` declaration might expand to include statements & configuration about the sub-module, I see no reason to prohibit Swift code from existing in that sub-module declaration file… Disallowing/controlling that seems to be a style/linter concern.

However, as I mentioned above, all the specific spellings (except `internal`) for the different concepts in this proposal are straw-men awaiting input. I’d say the addition of a new type of file extension raises some concerns for me, but there’s already been a lot of push back on the general idea of using filesystem structures to organize sub-modules, so the whole idea may be moot.

I’ve actually been starting to come around to the idea of using the file system. Not so much because I really like it, but because I have been considering further some of the drawbacks of other approaches.

One big reason is that a submodule should form a scope and scopes should consist of code that is physically adjacent. In practice this means it should reside in close proximity in the file system. Allowing any file in a project to be a part of any submodule partly defeats the purpose of using them to structure a project internally. If we’re going to be organizing the files in a submodule physically anyway maybe we should just take advantage of that fact and prevent a stray file in a distant part of the file system from being part of the submodule.

The second reason is that there is a big problem with placing a simple `submodule Foo` declaration at the top of each file in a submodule. We all make typos from time to time. This method makes it too easy to accidentally type `submodule Fooo` and end up in a submodule you didn’t intend. This mistake would likely be caught relatively quickly but it seems silly to have a system subject to this kind of mistake. This means we would end arbitrarily placing the declaration in one file and saying `extends submodule Foo` in the rest of the files. Your design avoids the need to arbitrarily choose where to place the declaration and avoids the need for a declaration in the rest of the files.

I’m not dead-set on this approach, but as you say, it solves a *lot* of problems that other approaches introduce. I do recognize the reasonableness of the main argument against, that file location shouldn’t have such a dramatic affect on behavior… *but* the fact is (whether by convention or expediency) we already *do* have filesystem location dependencies on our code projects…

* One can’t willy-nilly move files around the disk and expect a project to build… if you move it somewhere the compiler doesn’t know to look for it, you’re going to break things
* One can’t just move a file out of your SCM repo root, or you’ll “lose” the file

One might argue that these are pathological cases, but the fact remains that this approach only introduces a new constraint on project structure, not a new *kind* of constraint, and it is one that is purely optional. You don’t *have* to use sub-modules in a project, and if you don’t, there’s no additive burden on you.

Shortly, I’m going to update the original proposal gist reflecting all the comments I’ve received and part of that will be expanding the proposal section to include multiple alternate strategies for discussion, along with pros/cons for each. I’ll include a discussion on naming options for this file in this section.

I’m going to look at your new draft this afternoon while I’m working on updating my proposal. I think you’ll find it much more to your liking than the original draft. Perhaps we will be able to converge if we keep the discussion going.

I’d be happy to cooperate-with/converge-with/withdraw-in-favor-of. I’m mostly interested in a solid, light-weight solution to my use-cases, however that looks. I look forward to seeing your updates!

Sounds good. I’m hoping to have a new draft out later today.

If we’re going to use the file system to organize submodules this seems like a reasonable approach. It allows larger submodules to have folder hierarchies within them and also creates a central location for submodule-related declarations.

A primary flaw I see in this approach is that Xcode is the dominant IDE for Swift and the way Xcode handles files is not conducive to file-system organization. I really detest the way Xcode handles this and would vastly prefer that it simply reflected the physical file system hierarchy but I don’t think that will change any time soon. On the other hand maybe a file system based submodule system in Swift would motivate the Xcode team to better reflect the physical file system organization.

My thoughts (hopes?) on this were that should Swift adopt such a mechanism for sub-modules, then part of Xcode’s integration of the feature would be to make it aware of sub-modules in a first-class way, so that the author wouldn’t have to manually manage the project files to take advantage of the feature.

I’d envisioned either a new “Makes Sub-Module” property on Groups or a new “Sub-Module" mechanism alongside Groups & Folders in the project list. This may be wishful thinking on my part.

The problem with a "Makes Sub-Module” on a group in Xcode is that groups do not mirror the file system. Maybe a file system based submodule feature would motivate the Xcode team to support group structures that do mirror the file system better. I hope that would be the case! Unfortunately I don’t think we’d have a guarantee when reviewing the feature and there are significant practical implications for using the feature in Xcode without that kind of support. We would probably just have to trust Apple to get this right.

Too true. I don’t know how much the Swift team can influence or contribute to Xcode. On the other hand, using Xcode’s “Folder References” feature isn’t so onerous that it would prevent me from getting very large benefit from sub-modules in my projects.

I’ve always run into problems when I’ve tried to use folder references for code (they have worked find for bundle assets). It’s been a while since I’ve tried and I don’t remember exactly what the issues were. Maybe they’ve been solved (or maybe I was just doing something wrong).

—

As one concrete example, I have a large app that imports a Module (Framework) which represents a kind of domain-specific DOM graph. There is a Query type which must live within that module (both for WMO, and for more esoteric dependency reasons). This Query type has 11 related or sub- types including specialized graph nodes and a parser. Out of these 12 types, the API genuinely exposes 2 types (Query and QueryError) and a grand total of 5 methods/properties outside of the “Query” cluster of types. In order to achieve this encapsulation, I’ve had to lump all of these into a single 2k+ lines-long file with heavy use of fileprivate, which is a real pain to manage in terms of reading, editing, and SCM.

This would be a pain for sure. I try to keep files under 500 lines and usually shorter when possible.

One might say, oh, that’s not so bad, except this is one example of many similar situations in a very large codebase consisting of the Application and 9 subordinate Modules.

With this or similar sub-module system, this code could easily be factored into a tight cluster of specialized files, with no one file being larger than a couple of hundred lines, and most being in the 10–50 line neighborhood—much more manageable.

Supporting this use case well should be a primary goal for submodules.

In fact, I’d go so far as to say that if I had a real sub-module system, many of those Modules would get demoted. The cost of maintaining full frameworks for code that will never be used outside of this application just to achieve encapsulation isn’t trivial.

Yeah, submodules should eliminate the need for separate build products when you’re not ever deploying the code separately.

And another neat thing about a directory-based approach as I’ve proposed is that if I ever do want to “promote” a sub-module to its own Module, it’s very easy to do because I know where *all* the files are, and all of the sub-module’s child sub-modules automatically get “promoted up” one level without any effort.

Very true. And by keeping the submodule declaration out of the files they would not need to be edited to have the declarations in the files become top-level module declarations when they are moved to an independent module.

···

On Mar 5, 2017, at 8:52 AM, Karim Nassar <karim@karimnassar.com> wrote:

On Mar 4, 2017, at 2:54 PM, Matthew Johnson <matthew@anandabits.com <mailto:matthew@anandabits.com>> wrote:

On Mar 4, 2017, at 9:56 AM, Karim Nassar <karim@karimnassar.com <mailto:karim@karimnassar.com>> wrote:

On Mar 3, 2017, at 5:21 PM, Matthew Johnson <matthew@anandabits.com <mailto:matthew@anandabits.com>> wrote:

On Mar 3, 2017, at 9:24 AM, Karim Nassar via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

—Karim

16

Changes that *are* necessary are:

* Change the spelling of `internal` to `module` (making `module` the new default)
* Introduce a new modifier `internal` to mean "Internal to the current sub-module and its child-sub-modules”

Can you give concrete examples of use cases where a descendent submodule needs access to symbols declared by an ancestor? I gave some thought to this when drafting my proposal and came to the conclusion that this runs against the grain of layering and is likely to be a bad idea in practice. If there are use cases I didn’t consider I am very interested in learning about them.

On further reflection and examination of my notes, I think you’re right, and that the `internal` encapsulation should be purely horizontal. Will adjust to reflect that.

### Making a Sub-module

To create a sub-module within a Module (or sub-module) is simple: The author creates a directory, and places a "sub-module declaration file" within the directory:

```
// __submodule.swift

Why the double underscore prefix? To make it sort to the top in a file browser?

Is this file allowed to have any Swift code? Or is it limited to submodule-related declarations only? If the latter, why not use an extension such as `.submodule` or `.swiftmodule` to differentiate it from ordinary Swift files and allow the submodule to be named by the name of this file?

So, my reasoning was that by requiring a specific standard name for the declaration file, we guarantee that any directory can only describe one submodule. Prefixing the proposed name with underscores was simply a way of preventing naming collision with potential “real code” files (and yes, as a side-effect alpha-floating it to the top). Since the `submodule` declaration might expand to include statements & configuration about the sub-module, I see no reason to prohibit Swift code from existing in that sub-module declaration file… Disallowing/controlling that seems to be a style/linter concern.

However, as I mentioned above, all the specific spellings (except `internal`) for the different concepts in this proposal are straw-men awaiting input. I’d say the addition of a new type of file extension raises some concerns for me, but there’s already been a lot of push back on the general idea of using filesystem structures to organize sub-modules, so the whole idea may be moot.

I’ve actually been starting to come around to the idea of using the file system. Not so much because I really like it, but because I have been considering further some of the drawbacks of other approaches.

One big reason is that a submodule should form a scope and scopes should consist of code that is physically adjacent. In practice this means it should reside in close proximity in the file system. Allowing any file in a project to be a part of any submodule partly defeats the purpose of using them to structure a project internally. If we’re going to be organizing the files in a submodule physically anyway maybe we should just take advantage of that fact and prevent a stray file in a distant part of the file system from being part of the submodule.

The second reason is that there is a big problem with placing a simple `submodule Foo` declaration at the top of each file in a submodule. We all make typos from time to time. This method makes it too easy to accidentally type `submodule Fooo` and end up in a submodule you didn’t intend. This mistake would likely be caught relatively quickly but it seems silly to have a system subject to this kind of mistake. This means we would end arbitrarily placing the declaration in one file and saying `extends submodule Foo` in the rest of the files. Your design avoids the need to arbitrarily choose where to place the declaration and avoids the need for a declaration in the rest of the files.

I’m not dead-set on this approach, but as you say, it solves a *lot* of problems that other approaches introduce. I do recognize the reasonableness of the main argument against, that file location shouldn’t have such a dramatic affect on behavior… *but* the fact is (whether by convention or expediency) we already *do* have filesystem location dependencies on our code projects…

* One can’t willy-nilly move files around the disk and expect a project to build… if you move it somewhere the compiler doesn’t know to look for it, you’re going to break things
* One can’t just move a file out of your SCM repo root, or you’ll “lose” the file

True, but if other files do not refer to the lost file, you don’t even know for which file to look.
(imo this is already a weak point in swift, but this approach to submodules would make it -much?- worse)

If you were to include filenames in the “submodule declaration file” at least this omission in swift would be partly covered, and you would not need to move the other files into special places. (You could though)

···

On 05 Mar 2017, at 15:52, Karim Nassar via swift-evolution <swift-evolution@swift.org> wrote:

On Mar 4, 2017, at 2:54 PM, Matthew Johnson <matthew@anandabits.com> wrote:

On Mar 4, 2017, at 9:56 AM, Karim Nassar <karim@karimnassar.com> wrote:

On Mar 3, 2017, at 5:21 PM, Matthew Johnson <matthew@anandabits.com> wrote:

On Mar 3, 2017, at 9:24 AM, Karim Nassar via swift-evolution <swift-evolution@swift.org> wrote:

One might argue that these are pathological cases, but the fact remains that this approach only introduces a new constraint on project structure, not a new *kind* of constraint, and it is one that is purely optional. You don’t *have* to use sub-modules in a project, and if you don’t, there’s no additive burden on you.

Shortly, I’m going to update the original proposal gist reflecting all the comments I’ve received and part of that will be expanding the proposal section to include multiple alternate strategies for discussion, along with pros/cons for each. I’ll include a discussion on naming options for this file in this section.

I’m going to look at your new draft this afternoon while I’m working on updating my proposal. I think you’ll find it much more to your liking than the original draft. Perhaps we will be able to converge if we keep the discussion going.

I’d be happy to cooperate-with/converge-with/withdraw-in-favor-of. I’m mostly interested in a solid, light-weight solution to my use-cases, however that looks. I look forward to seeing your updates!

If we’re going to use the file system to organize submodules this seems like a reasonable approach. It allows larger submodules to have folder hierarchies within them and also creates a central location for submodule-related declarations.

A primary flaw I see in this approach is that Xcode is the dominant IDE for Swift and the way Xcode handles files is not conducive to file-system organization. I really detest the way Xcode handles this and would vastly prefer that it simply reflected the physical file system hierarchy but I don’t think that will change any time soon. On the other hand maybe a file system based submodule system in Swift would motivate the Xcode team to better reflect the physical file system organization.

My thoughts (hopes?) on this were that should Swift adopt such a mechanism for sub-modules, then part of Xcode’s integration of the feature would be to make it aware of sub-modules in a first-class way, so that the author wouldn’t have to manually manage the project files to take advantage of the feature.

I’d envisioned either a new “Makes Sub-Module” property on Groups or a new “Sub-Module" mechanism alongside Groups & Folders in the project list. This may be wishful thinking on my part.

The problem with a "Makes Sub-Module” on a group in Xcode is that groups do not mirror the file system. Maybe a file system based submodule feature would motivate the Xcode team to support group structures that do mirror the file system better. I hope that would be the case! Unfortunately I don’t think we’d have a guarantee when reviewing the feature and there are significant practical implications for using the feature in Xcode without that kind of support. We would probably just have to trust Apple to get this right.

Too true. I don’t know how much the Swift team can influence or contribute to Xcode. On the other hand, using Xcode’s “Folder References” feature isn’t so onerous that it would prevent me from getting very large benefit from sub-modules in my projects.

—

As one concrete example, I have a large app that imports a Module (Framework) which represents a kind of domain-specific DOM graph. There is a Query type which must live within that module (both for WMO, and for more esoteric dependency reasons). This Query type has 11 related or sub- types including specialized graph nodes and a parser. Out of these 12 types, the API genuinely exposes 2 types (Query and QueryError) and a grand total of 5 methods/properties outside of the “Query” cluster of types. In order to achieve this encapsulation, I’ve had to lump all of these into a single 2k+ lines-long file with heavy use of fileprivate, which is a real pain to manage in terms of reading, editing, and SCM.

One might say, oh, that’s not so bad, except this is one example of many similar situations in a very large codebase consisting of the Application and 9 subordinate Modules.

With this or similar sub-module system, this code could easily be factored into a tight cluster of specialized files, with no one file being larger than a couple of hundred lines, and most being in the 10–50 line neighborhood—much more manageable.

In fact, I’d go so far as to say that if I had a real sub-module system, many of those Modules would get demoted. The cost of maintaining full frameworks for code that will never be used outside of this application just to achieve encapsulation isn’t trivial.

And another neat thing about a directory-based approach as I’ve proposed is that if I ever do want to “promote” a sub-module to its own Module, it’s very easy to do because I know where *all* the files are, and all of the sub-module’s child sub-modules automatically get “promoted up” one level without any effort.

—Karim

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

17

Changes that *are* necessary are:

* Change the spelling of `internal` to `module` (making `module` the new default)
* Introduce a new modifier `internal` to mean "Internal to the current sub-module and its child-sub-modules”

Can you give concrete examples of use cases where a descendent submodule needs access to symbols declared by an ancestor? I gave some thought to this when drafting my proposal and came to the conclusion that this runs against the grain of layering and is likely to be a bad idea in practice. If there are use cases I didn’t consider I am very interested in learning about them.

On further reflection and examination of my notes, I think you’re right, and that the `internal` encapsulation should be purely horizontal. Will adjust to reflect that.

### Making a Sub-module

To create a sub-module within a Module (or sub-module) is simple: The author creates a directory, and places a "sub-module declaration file" within the directory:

```
// __submodule.swift

Why the double underscore prefix? To make it sort to the top in a file browser?

Is this file allowed to have any Swift code? Or is it limited to submodule-related declarations only? If the latter, why not use an extension such as `.submodule` or `.swiftmodule` to differentiate it from ordinary Swift files and allow the submodule to be named by the name of this file?

So, my reasoning was that by requiring a specific standard name for the declaration file, we guarantee that any directory can only describe one submodule. Prefixing the proposed name with underscores was simply a way of preventing naming collision with potential “real code” files (and yes, as a side-effect alpha-floating it to the top). Since the `submodule` declaration might expand to include statements & configuration about the sub-module, I see no reason to prohibit Swift code from existing in that sub-module declaration file… Disallowing/controlling that seems to be a style/linter concern.

However, as I mentioned above, all the specific spellings (except `internal`) for the different concepts in this proposal are straw-men awaiting input. I’d say the addition of a new type of file extension raises some concerns for me, but there’s already been a lot of push back on the general idea of using filesystem structures to organize sub-modules, so the whole idea may be moot.

I’ve actually been starting to come around to the idea of using the file system. Not so much because I really like it, but because I have been considering further some of the drawbacks of other approaches.

One big reason is that a submodule should form a scope and scopes should consist of code that is physically adjacent. In practice this means it should reside in close proximity in the file system. Allowing any file in a project to be a part of any submodule partly defeats the purpose of using them to structure a project internally. If we’re going to be organizing the files in a submodule physically anyway maybe we should just take advantage of that fact and prevent a stray file in a distant part of the file system from being part of the submodule.

The second reason is that there is a big problem with placing a simple `submodule Foo` declaration at the top of each file in a submodule. We all make typos from time to time. This method makes it too easy to accidentally type `submodule Fooo` and end up in a submodule you didn’t intend. This mistake would likely be caught relatively quickly but it seems silly to have a system subject to this kind of mistake. This means we would end arbitrarily placing the declaration in one file and saying `extends submodule Foo` in the rest of the files. Your design avoids the need to arbitrarily choose where to place the declaration and avoids the need for a declaration in the rest of the files.

I’m not dead-set on this approach, but as you say, it solves a *lot* of problems that other approaches introduce. I do recognize the reasonableness of the main argument against, that file location shouldn’t have such a dramatic affect on behavior… *but* the fact is (whether by convention or expediency) we already *do* have filesystem location dependencies on our code projects…

* One can’t willy-nilly move files around the disk and expect a project to build… if you move it somewhere the compiler doesn’t know to look for it, you’re going to break things
* One can’t just move a file out of your SCM repo root, or you’ll “lose” the file

True, but if other files do not refer to the lost file, you don’t even know for which file to look.
(imo this is already a weak point in swift, but this approach to submodules would make it -much?- worse)

If you were to include filenames in the “submodule declaration file” at least this omission in swift would be partly covered, and you would not need to move the other files into special places. (You could though)

Listing the file names is what I call the “manifest” approach. I think it would b pretty painful to maintain. It’s definitely not a lightweight solution. I don’t think the “lost file” use case should be a primary motivation behind the design we adopt.

I think it’s fair to say that helping users recover from a lost file is a problem that should be solved at a higher level by tools (Xcode helps with this for example). I don’t think it should influence the design of the language.

···

On Mar 5, 2017, at 9:10 AM, Rien <Rien@Balancingrock.nl> wrote:

On 05 Mar 2017, at 15:52, Karim Nassar via swift-evolution <swift-evolution@swift.org> wrote:

On Mar 4, 2017, at 2:54 PM, Matthew Johnson <matthew@anandabits.com> wrote:

On Mar 4, 2017, at 9:56 AM, Karim Nassar <karim@karimnassar.com> wrote:

On Mar 3, 2017, at 5:21 PM, Matthew Johnson <matthew@anandabits.com> wrote:

On Mar 3, 2017, at 9:24 AM, Karim Nassar via swift-evolution <swift-evolution@swift.org> wrote:

One might argue that these are pathological cases, but the fact remains that this approach only introduces a new constraint on project structure, not a new *kind* of constraint, and it is one that is purely optional. You don’t *have* to use sub-modules in a project, and if you don’t, there’s no additive burden on you.

Shortly, I’m going to update the original proposal gist reflecting all the comments I’ve received and part of that will be expanding the proposal section to include multiple alternate strategies for discussion, along with pros/cons for each. I’ll include a discussion on naming options for this file in this section.

I’m going to look at your new draft this afternoon while I’m working on updating my proposal. I think you’ll find it much more to your liking than the original draft. Perhaps we will be able to converge if we keep the discussion going.

I’d be happy to cooperate-with/converge-with/withdraw-in-favor-of. I’m mostly interested in a solid, light-weight solution to my use-cases, however that looks. I look forward to seeing your updates!

If we’re going to use the file system to organize submodules this seems like a reasonable approach. It allows larger submodules to have folder hierarchies within them and also creates a central location for submodule-related declarations.

A primary flaw I see in this approach is that Xcode is the dominant IDE for Swift and the way Xcode handles files is not conducive to file-system organization. I really detest the way Xcode handles this and would vastly prefer that it simply reflected the physical file system hierarchy but I don’t think that will change any time soon. On the other hand maybe a file system based submodule system in Swift would motivate the Xcode team to better reflect the physical file system organization.

My thoughts (hopes?) on this were that should Swift adopt such a mechanism for sub-modules, then part of Xcode’s integration of the feature would be to make it aware of sub-modules in a first-class way, so that the author wouldn’t have to manually manage the project files to take advantage of the feature.

I’d envisioned either a new “Makes Sub-Module” property on Groups or a new “Sub-Module" mechanism alongside Groups & Folders in the project list. This may be wishful thinking on my part.

The problem with a "Makes Sub-Module” on a group in Xcode is that groups do not mirror the file system. Maybe a file system based submodule feature would motivate the Xcode team to support group structures that do mirror the file system better. I hope that would be the case! Unfortunately I don’t think we’d have a guarantee when reviewing the feature and there are significant practical implications for using the feature in Xcode without that kind of support. We would probably just have to trust Apple to get this right.

Too true. I don’t know how much the Swift team can influence or contribute to Xcode. On the other hand, using Xcode’s “Folder References” feature isn’t so onerous that it would prevent me from getting very large benefit from sub-modules in my projects.

—

As one concrete example, I have a large app that imports a Module (Framework) which represents a kind of domain-specific DOM graph. There is a Query type which must live within that module (both for WMO, and for more esoteric dependency reasons). This Query type has 11 related or sub- types including specialized graph nodes and a parser. Out of these 12 types, the API genuinely exposes 2 types (Query and QueryError) and a grand total of 5 methods/properties outside of the “Query” cluster of types. In order to achieve this encapsulation, I’ve had to lump all of these into a single 2k+ lines-long file with heavy use of fileprivate, which is a real pain to manage in terms of reading, editing, and SCM.

One might say, oh, that’s not so bad, except this is one example of many similar situations in a very large codebase consisting of the Application and 9 subordinate Modules.

With this or similar sub-module system, this code could easily be factored into a tight cluster of specialized files, with no one file being larger than a couple of hundred lines, and most being in the 10–50 line neighborhood—much more manageable.

In fact, I’d go so far as to say that if I had a real sub-module system, many of those Modules would get demoted. The cost of maintaining full frameworks for code that will never be used outside of this application just to achieve encapsulation isn’t trivial.

And another neat thing about a directory-based approach as I’ve proposed is that if I ever do want to “promote” a sub-module to its own Module, it’s very easy to do because I know where *all* the files are, and all of the sub-module’s child sub-modules automatically get “promoted up” one level without any effort.

—Karim

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

18

But isn’t this exactly the same case if you’re using Swift without an IDE? It’s true that Xcode keeps track of files in your project, and there’s no reason it couldn’t also keep track of files in a submodule (in fact it would… it just might not know they belong in the sub-module without some work from the Xcode dev team).

But as a thought-experiment, here’s how I’m thinking of this problem:

Swift (as a language) has established that a file *is* a unit of code. Not the only one of course, but it is one, as evidenced by access controls (fileprivate, Swift2 private) which bound on the file.

I (and many others) want to be able to describe another unit of code larger than a file and smaller than a Module to help organize the file-units we have which go together within a Module, but which shouldn’t (for various reasons) describe a complete Module in and of themselves.

For the moment—though we actually propose to call this unit a sub-module—we'll call this new unit a Foo. Let's describe properties of a Foo:

* A Foo should serve to organize one or more File code units by:
  - ensuring a File (the code-unit) may belong to one and only one Foo
  - making clear (with or without an IDE) to which Foo a given file might belong
* A Foo should have a unique name
* A Foo should be able to nest under another Foo, and the name of that nesting should be clear & unique
* A Foo should be able to be easily created, split, and merged as the codebase evolves
* A Foo should be able to be moved, copied, and SCMed as a unit, without reference to anything outside the Foo

Having now described the critical properties of this new unit called “Foo” have we not also described a filesystem directory?

Naturally, we don’t want *all* directories to become Foos (that would be overly restrictive to authors, and backwards-incompatible), so we need a way to distinguish a normal directory from one that is acting as a Foo. I originally proposed a special “sub-module declaration file”, but there may be other ways. For example, we might establish that a directory whose name matches `[SubModuleName].swift-sub-module` becomes a sub-module of designated name, however this can cause problems due to the disjoint sets of “Legal Filesystem Directory Characters” and “Legal Swift Identifier Characters”. It also doesn’t give us a place to “grow” the declaration in the future if we choose to add additional properties to the sub-module declaration.

—Karim

···

On Mar 5, 2017, at 10:10 AM, Rien <Rien@Balancingrock.nl> wrote:

On 05 Mar 2017, at 15:52, Karim Nassar via swift-evolution <swift-evolution@swift.org> wrote:

On Mar 4, 2017, at 2:54 PM, Matthew Johnson <matthew@anandabits.com> wrote:

On Mar 4, 2017, at 9:56 AM, Karim Nassar <karim@karimnassar.com> wrote:

On Mar 3, 2017, at 5:21 PM, Matthew Johnson <matthew@anandabits.com> wrote:

On Mar 3, 2017, at 9:24 AM, Karim Nassar via swift-evolution <swift-evolution@swift.org> wrote:

Changes that *are* necessary are:

* Change the spelling of `internal` to `module` (making `module` the new default)
* Introduce a new modifier `internal` to mean "Internal to the current sub-module and its child-sub-modules”

Can you give concrete examples of use cases where a descendent submodule needs access to symbols declared by an ancestor? I gave some thought to this when drafting my proposal and came to the conclusion that this runs against the grain of layering and is likely to be a bad idea in practice. If there are use cases I didn’t consider I am very interested in learning about them.

On further reflection and examination of my notes, I think you’re right, and that the `internal` encapsulation should be purely horizontal. Will adjust to reflect that.

### Making a Sub-module

To create a sub-module within a Module (or sub-module) is simple: The author creates a directory, and places a "sub-module declaration file" within the directory:

```
// __submodule.swift

Why the double underscore prefix? To make it sort to the top in a file browser?

Is this file allowed to have any Swift code? Or is it limited to submodule-related declarations only? If the latter, why not use an extension such as `.submodule` or `.swiftmodule` to differentiate it from ordinary Swift files and allow the submodule to be named by the name of this file?

So, my reasoning was that by requiring a specific standard name for the declaration file, we guarantee that any directory can only describe one submodule. Prefixing the proposed name with underscores was simply a way of preventing naming collision with potential “real code” files (and yes, as a side-effect alpha-floating it to the top). Since the `submodule` declaration might expand to include statements & configuration about the sub-module, I see no reason to prohibit Swift code from existing in that sub-module declaration file… Disallowing/controlling that seems to be a style/linter concern.

However, as I mentioned above, all the specific spellings (except `internal`) for the different concepts in this proposal are straw-men awaiting input. I’d say the addition of a new type of file extension raises some concerns for me, but there’s already been a lot of push back on the general idea of using filesystem structures to organize sub-modules, so the whole idea may be moot.

I’ve actually been starting to come around to the idea of using the file system. Not so much because I really like it, but because I have been considering further some of the drawbacks of other approaches.

One big reason is that a submodule should form a scope and scopes should consist of code that is physically adjacent. In practice this means it should reside in close proximity in the file system. Allowing any file in a project to be a part of any submodule partly defeats the purpose of using them to structure a project internally. If we’re going to be organizing the files in a submodule physically anyway maybe we should just take advantage of that fact and prevent a stray file in a distant part of the file system from being part of the submodule.

The second reason is that there is a big problem with placing a simple `submodule Foo` declaration at the top of each file in a submodule. We all make typos from time to time. This method makes it too easy to accidentally type `submodule Fooo` and end up in a submodule you didn’t intend. This mistake would likely be caught relatively quickly but it seems silly to have a system subject to this kind of mistake. This means we would end arbitrarily placing the declaration in one file and saying `extends submodule Foo` in the rest of the files. Your design avoids the need to arbitrarily choose where to place the declaration and avoids the need for a declaration in the rest of the files.

I’m not dead-set on this approach, but as you say, it solves a *lot* of problems that other approaches introduce. I do recognize the reasonableness of the main argument against, that file location shouldn’t have such a dramatic affect on behavior… *but* the fact is (whether by convention or expediency) we already *do* have filesystem location dependencies on our code projects…

* One can’t willy-nilly move files around the disk and expect a project to build… if you move it somewhere the compiler doesn’t know to look for it, you’re going to break things
* One can’t just move a file out of your SCM repo root, or you’ll “lose” the file

True, but if other files do not refer to the lost file, you don’t even know for which file to look.
(imo this is already a weak point in swift, but this approach to submodules would make it -much?- worse)

If you were to include filenames in the “submodule declaration file” at least this omission in swift would be partly covered, and you would not need to move the other files into special places. (You could though)

19

Punycode would readily cover your first issue about legal identifier
characters.

As to the second, having enumerated the essential properties of a Foo, what
"growth" are you envisioning?

···

On Sun, Mar 5, 2017 at 12:51 Karim Nassar via swift-evolution < swift-evolution@swift.org> wrote:

On Mar 5, 2017, at 10:10 AM, Rien <Rien@Balancingrock.nl> wrote:

On 05 Mar 2017, at 15:52, Karim Nassar via swift-evolution < > swift-evolution@swift.org> wrote:

On Mar 4, 2017, at 2:54 PM, Matthew Johnson <matthew@anandabits.com> > wrote:

On Mar 4, 2017, at 9:56 AM, Karim Nassar <karim@karimnassar.com> wrote:

On Mar 3, 2017, at 5:21 PM, Matthew Johnson <matthew@anandabits.com> > wrote:

On Mar 3, 2017, at 9:24 AM, Karim Nassar via swift-evolution < > swift-evolution@swift.org> wrote:

Changes that *are* necessary are:

* Change the spelling of `internal` to `module` (making `module` the new
default)
* Introduce a new modifier `internal` to mean "Internal to the current
sub-module and its child-sub-modules”

Can you give concrete examples of use cases where a descendent submodule
needs access to symbols declared by an ancestor? I gave some thought to
this when drafting my proposal and came to the conclusion that this runs
against the grain of layering and is likely to be a bad idea in practice.
If there are use cases I didn’t consider I am very interested in learning
about them.

On further reflection and examination of my notes, I think you’re right,
and that the `internal` encapsulation should be purely horizontal. Will
adjust to reflect that.

### Making a Sub-module

To create a sub-module within a Module (or sub-module) is simple: The
author creates a directory, and places a "sub-module declaration file"
within the directory:

```
// __submodule.swift

Why the double underscore prefix? To make it sort to the top in a file
browser?

Is this file allowed to have any Swift code? Or is it limited to
submodule-related declarations only? If the latter, why not use an
extension such as `.submodule` or `.swiftmodule` to differentiate it from
ordinary Swift files and allow the submodule to be named by the name of
this file?

So, my reasoning was that by requiring a specific standard name for the
declaration file, we guarantee that any directory can only describe one
submodule. Prefixing the proposed name with underscores was simply a way of
preventing naming collision with potential “real code” files (and yes, as a
side-effect alpha-floating it to the top). Since the `submodule`
declaration might expand to include statements & configuration about the
sub-module, I see no reason to prohibit Swift code from existing in that
sub-module declaration file… Disallowing/controlling that seems to be a
style/linter concern.

However, as I mentioned above, all the specific spellings (except
`internal`) for the different concepts in this proposal are straw-men
awaiting input. I’d say the addition of a new type of file extension raises
some concerns for me, but there’s already been a lot of push back on the
general idea of using filesystem structures to organize sub-modules, so the
whole idea may be moot.

I’ve actually been starting to come around to the idea of using the file
system. Not so much because I really like it, but because I have been
considering further some of the drawbacks of other approaches.

One big reason is that a submodule should form a scope and scopes should
consist of code that is physically adjacent. In practice this means it
should reside in close proximity in the file system. Allowing any file in
a project to be a part of any submodule partly defeats the purpose of using
them to structure a project internally. If we’re going to be organizing
the files in a submodule physically anyway maybe we should just take
advantage of that fact and prevent a stray file in a distant part of the
file system from being part of the submodule.

The second reason is that there is a big problem with placing a simple
`submodule Foo` declaration at the top of each file in a submodule. We all
make typos from time to time. This method makes it too easy to
accidentally type `submodule Fooo` and end up in a submodule you didn’t
intend. This mistake would likely be caught relatively quickly but it
seems silly to have a system subject to this kind of mistake. This means
we would end arbitrarily placing the declaration in one file and saying
`extends submodule Foo` in the rest of the files. Your design avoids the
need to arbitrarily choose where to place the declaration and avoids the
need for a declaration in the rest of the files.

I’m not dead-set on this approach, but as you say, it solves a *lot* of
problems that other approaches introduce. I do recognize the reasonableness
of the main argument against, that file location shouldn’t have such a
dramatic affect on behavior… *but* the fact is (whether by convention or
expediency) we already *do* have filesystem location dependencies on our
code projects…

* One can’t willy-nilly move files around the disk and expect a project to
build… if you move it somewhere the compiler doesn’t know to look for it,
you’re going to break things
* One can’t just move a file out of your SCM repo root, or you’ll “lose”
the file

True, but if other files do not refer to the lost file, you don’t even
know for which file to look.
(imo this is already a weak point in swift, but this approach to
submodules would make it -much?- worse)

If you were to include filenames in the “submodule declaration file” at
least this omission in swift would be partly covered, and you would not
need to move the other files into special places. (You could though)

But isn’t this exactly the same case if you’re using Swift without an IDE?
It’s true that Xcode keeps track of files in your project, and there’s no
reason it couldn’t also keep track of files in a submodule (in fact it
would… it just might not know they belong in the sub-module without some
work from the Xcode dev team).

But as a thought-experiment, here’s how I’m thinking of this problem:

Swift (as a language) has established that a file *is* a unit of code. Not
the only one of course, but it is one, as evidenced by access controls
(fileprivate, Swift2 private) which bound on the file.

I (and many others) want to be able to describe another unit of code
larger than a file and smaller than a Module to help organize the
file-units we have which go together within a Module, but which shouldn’t
(for various reasons) describe a complete Module in and of themselves.

For the moment—though we actually propose to call this unit a
sub-module—we'll call this new unit a Foo. Let's describe properties of a
Foo:

* A Foo should serve to organize one or more File code units by:
  - ensuring a File (the code-unit) may belong to one and only one Foo
  - making clear (with or without an IDE) to which Foo a given file might
belong
* A Foo should have a unique name
* A Foo should be able to nest under another Foo, and the name of that
nesting should be clear & unique
* A Foo should be able to be easily created, split, and merged as the
codebase evolves
* A Foo should be able to be moved, copied, and SCMed as a unit, without
reference to anything outside the Foo

Having now described the critical properties of this new unit called “Foo”
have we not also described a filesystem directory?

Naturally, we don’t want *all* directories to become Foos (that would be
overly restrictive to authors, and backwards-incompatible), so we need a
way to distinguish a normal directory from one that is acting as a Foo. I
originally proposed a special “sub-module declaration file”, but there may
be other ways. For example, we might establish that a directory whose name
matches `[SubModuleName].swift-sub-module` becomes a sub-module of
designated name, however this can cause problems due to the disjoint sets
of “Legal Filesystem Directory Characters” and “Legal Swift Identifier
Characters”. It also doesn’t give us a place to “grow” the declaration in
the future if we choose to add additional properties to the sub-module
declaration.

—Karim

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

20

I’m not dead-set on this approach, but as you say, it solves a *lot* of problems that other approaches introduce. I do recognize the reasonableness of the main argument against, that file location shouldn’t have such a dramatic affect on behavior… *but* the fact is (whether by convention or expediency) we already *do* have filesystem location dependencies on our code projects…

* One can’t willy-nilly move files around the disk and expect a project to build… if you move it somewhere the compiler doesn’t know to look for it, you’re going to break things
* One can’t just move a file out of your SCM repo root, or you’ll “lose” the file

True, but if other files do not refer to the lost file, you don’t even know for which file to look.
(imo this is already a weak point in swift, but this approach to submodules would make it -much?- worse)

If you were to include filenames in the “submodule declaration file” at least this omission in swift would be partly covered, and you would not need to move the other files into special places. (You could though)

But isn’t this exactly the same case if you’re using Swift without an IDE? It’s true that Xcode keeps track of files in your project, and there’s no reason it couldn’t also keep track of files in a submodule (in fact it would… it just might not know they belong in the sub-module without some work from the Xcode dev team).

But as a thought-experiment, here’s how I’m thinking of this problem:

Swift (as a language) has established that a file *is* a unit of code. Not the only one of course, but it is one, as evidenced by access controls (fileprivate, Swift2 private) which bound on the file.

I disagree with this: Swift is just a set of -more or less- arbitrary rules to allow the development of applications.
The criteria for those rules are chosen by the core team and will fluctuate with time.
As such there is no “Swift way” of doing things. Any such remark is by its nature a personal preference. (not that there is anything wrong with that, its just not an argument)

(Besides, a computer only knows files, thus of course a file is a unit of code. That does not mean we should extend “code” to be something outside the files)

I (and many others) want to be able to describe another unit of code larger than a file and smaller than a Module to help organize the file-units we have which go together within a Module, but which shouldn’t (for various reasons) describe a complete Module in and of themselves.

Sure, and I have nothing against that. However I an dead set against extending the scope of the language itself to the organisation of its files. It is traditional to leave the organisation of the files to the configuration of the compiler/linker. And I do not see how it would simplify the language itself by extending the language to cover file organisation.

I will predict that if this approach is taken you should brace for the storm that would want to reverse this feature. It would imo dwarf the private vs fileprivate debate.

Besides, why create something new? (Yes there are systems out there that do some of this) There are many existing solutions out there that we could adopt and stay completely within the bounds of the language, and give the users something they are likely already familiar with.

For the moment—though we actually propose to call this unit a sub-module—we'll call this new unit a Foo. Let's describe properties of a Foo:

* A Foo should serve to organize one or more File code units by:
  - ensuring a File (the code-unit) may belong to one and only one Foo
  - making clear (with or without an IDE) to which Foo a given file might belong
* A Foo should have a unique name
* A Foo should be able to nest under another Foo, and the name of that nesting should be clear & unique
* A Foo should be able to be easily created, split, and merged as the codebase evolves
* A Foo should be able to be moved, copied, and SCMed as a unit, without reference to anything outside the Foo

Having now described the critical properties of this new unit called “Foo” have we not also described a filesystem directory?

Many things in life share characteristics. And while we should certainly look to them for lessons, we should not conclude that they are the same.

Rien.

···

Naturally, we don’t want *all* directories to become Foos (that would be overly restrictive to authors, and backwards-incompatible), so we need a way to distinguish a normal directory from one that is acting as a Foo. I originally proposed a special “sub-module declaration file”, but there may be other ways. For example, we might establish that a directory whose name matches `[SubModuleName].swift-sub-module` becomes a sub-module of designated name, however this can cause problems due to the disjoint sets of “Legal Filesystem Directory Characters” and “Legal Swift Identifier Characters”. It also doesn’t give us a place to “grow” the declaration in the future if we choose to add additional properties to the sub-module declaration.

—Karim