Yet Another Take on Swift Sub-modules

Punycode would readily cover your first issue about legal identifier characters.

As to the second, having enumerated the essential properties of a Foo, what "growth" are you envisioning?

I am working on an updated draft of my scope-based submodule proposal which I hope to complete this afternoon. It is significantly improved from the earlier draft. It allows submodule declarations to be decorated with a small number of very useful annotations. I encourage you to keep an eye out for the new draft which will explain why a simple directory naming scheme won’t be sufficient.

···

On Mar 5, 2017, at 1:06 PM, Xiaodi Wu via swift-evolution <swift-evolution@swift.org> wrote:

On Sun, Mar 5, 2017 at 12:51 Karim Nassar via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Mar 5, 2017, at 10:10 AM, Rien <Rien@Balancingrock.nl <mailto:Rien@Balancingrock.nl>> wrote:

On 05 Mar 2017, at 15:52, Karim Nassar via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Mar 4, 2017, at 2:54 PM, Matthew Johnson <matthew@anandabits.com <mailto:matthew@anandabits.com>> wrote:

On Mar 4, 2017, at 9:56 AM, Karim Nassar <karim@karimnassar.com <mailto:karim@karimnassar.com>> wrote:

On Mar 3, 2017, at 5:21 PM, Matthew Johnson <matthew@anandabits.com <mailto:matthew@anandabits.com>> wrote:

On Mar 3, 2017, at 9:24 AM, Karim Nassar via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Changes that *are* necessary are:

* Change the spelling of `internal` to `module` (making `module` the new default)
* Introduce a new modifier `internal` to mean "Internal to the current sub-module and its child-sub-modules”

Can you give concrete examples of use cases where a descendent submodule needs access to symbols declared by an ancestor? I gave some thought to this when drafting my proposal and came to the conclusion that this runs against the grain of layering and is likely to be a bad idea in practice. If there are use cases I didn’t consider I am very interested in learning about them.

On further reflection and examination of my notes, I think you’re right, and that the `internal` encapsulation should be purely horizontal. Will adjust to reflect that.

### Making a Sub-module

To create a sub-module within a Module (or sub-module) is simple: The author creates a directory, and places a "sub-module declaration file" within the directory:

```
// __submodule.swift

Why the double underscore prefix? To make it sort to the top in a file browser?

Is this file allowed to have any Swift code? Or is it limited to submodule-related declarations only? If the latter, why not use an extension such as `.submodule` or `.swiftmodule` to differentiate it from ordinary Swift files and allow the submodule to be named by the name of this file?

So, my reasoning was that by requiring a specific standard name for the declaration file, we guarantee that any directory can only describe one submodule. Prefixing the proposed name with underscores was simply a way of preventing naming collision with potential “real code” files (and yes, as a side-effect alpha-floating it to the top). Since the `submodule` declaration might expand to include statements & configuration about the sub-module, I see no reason to prohibit Swift code from existing in that sub-module declaration file… Disallowing/controlling that seems to be a style/linter concern.

However, as I mentioned above, all the specific spellings (except `internal`) for the different concepts in this proposal are straw-men awaiting input. I’d say the addition of a new type of file extension raises some concerns for me, but there’s already been a lot of push back on the general idea of using filesystem structures to organize sub-modules, so the whole idea may be moot.

I’ve actually been starting to come around to the idea of using the file system. Not so much because I really like it, but because I have been considering further some of the drawbacks of other approaches.

One big reason is that a submodule should form a scope and scopes should consist of code that is physically adjacent. In practice this means it should reside in close proximity in the file system. Allowing any file in a project to be a part of any submodule partly defeats the purpose of using them to structure a project internally. If we’re going to be organizing the files in a submodule physically anyway maybe we should just take advantage of that fact and prevent a stray file in a distant part of the file system from being part of the submodule.

The second reason is that there is a big problem with placing a simple `submodule Foo` declaration at the top of each file in a submodule. We all make typos from time to time. This method makes it too easy to accidentally type `submodule Fooo` and end up in a submodule you didn’t intend. This mistake would likely be caught relatively quickly but it seems silly to have a system subject to this kind of mistake. This means we would end arbitrarily placing the declaration in one file and saying `extends submodule Foo` in the rest of the files. Your design avoids the need to arbitrarily choose where to place the declaration and avoids the need for a declaration in the rest of the files.

I’m not dead-set on this approach, but as you say, it solves a *lot* of problems that other approaches introduce. I do recognize the reasonableness of the main argument against, that file location shouldn’t have such a dramatic affect on behavior… *but* the fact is (whether by convention or expediency) we already *do* have filesystem location dependencies on our code projects…

* One can’t willy-nilly move files around the disk and expect a project to build… if you move it somewhere the compiler doesn’t know to look for it, you’re going to break things
* One can’t just move a file out of your SCM repo root, or you’ll “lose” the file

True, but if other files do not refer to the lost file, you don’t even know for which file to look.
(imo this is already a weak point in swift, but this approach to submodules would make it -much?- worse)

If you were to include filenames in the “submodule declaration file” at least this omission in swift would be partly covered, and you would not need to move the other files into special places. (You could though)

But isn’t this exactly the same case if you’re using Swift without an IDE? It’s true that Xcode keeps track of files in your project, and there’s no reason it couldn’t also keep track of files in a submodule (in fact it would… it just might not know they belong in the sub-module without some work from the Xcode dev team).

But as a thought-experiment, here’s how I’m thinking of this problem:

Swift (as a language) has established that a file *is* a unit of code. Not the only one of course, but it is one, as evidenced by access controls (fileprivate, Swift2 private) which bound on the file.

I (and many others) want to be able to describe another unit of code larger than a file and smaller than a Module to help organize the file-units we have which go together within a Module, but which shouldn’t (for various reasons) describe a complete Module in and of themselves.

For the moment—though we actually propose to call this unit a sub-module—we'll call this new unit a Foo. Let's describe properties of a Foo:

* A Foo should serve to organize one or more File code units by:
  - ensuring a File (the code-unit) may belong to one and only one Foo
  - making clear (with or without an IDE) to which Foo a given file might belong
* A Foo should have a unique name
* A Foo should be able to nest under another Foo, and the name of that nesting should be clear & unique
* A Foo should be able to be easily created, split, and merged as the codebase evolves
* A Foo should be able to be moved, copied, and SCMed as a unit, without reference to anything outside the Foo

Having now described the critical properties of this new unit called “Foo” have we not also described a filesystem directory?

Naturally, we don’t want *all* directories to become Foos (that would be overly restrictive to authors, and backwards-incompatible), so we need a way to distinguish a normal directory from one that is acting as a Foo. I originally proposed a special “sub-module declaration file”, but there may be other ways. For example, we might establish that a directory whose name matches `[SubModuleName].swift-sub-module` becomes a sub-module of designated name, however this can cause problems due to the disjoint sets of “Legal Filesystem Directory Characters” and “Legal Swift Identifier Characters”. It also doesn’t give us a place to “grow” the declaration in the future if we choose to add additional properties to the sub-module declaration.

—Karim

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution
_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Punycode would readily cover your first issue about legal identifier characters.

Well, it would cover… I’m not sure if “readily” would be the right word.

If I have a sub-module I want to call `Bücher`, and I have to name the directory `Bcher-kva`, what do I use to refer to the sub-module in my code? It’s not at all readable, nor is it discoverable to a 3rd party who if they don’t “know” about Punycode, have no reason to associate the `Bücher` sub-module with the `Bcher-kva` directory.

Not that’s a show-stopper, mind. I wouldn’t mind further exploring the directory-name + “extension" approach.

If we explore this I think it needs to be as a form of “syntactic sugar” only. We still need a place to write an explicit submodule declaration.

···

On Mar 5, 2017, at 1:29 PM, Karim Nassar via swift-evolution <swift-evolution@swift.org> wrote:

On Mar 5, 2017, at 2:06 PM, Xiaodi Wu <xiaodi.wu@gmail.com <mailto:xiaodi.wu@gmail.com>> wrote:

—Karim

As to the second, having enumerated the essential properties of a Foo, what "growth" are you envisioning?
On Sun, Mar 5, 2017 at 12:51 Karim Nassar via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Mar 5, 2017, at 10:10 AM, Rien <Rien@Balancingrock.nl <mailto:Rien@Balancingrock.nl>> wrote:

On 05 Mar 2017, at 15:52, Karim Nassar via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Mar 4, 2017, at 2:54 PM, Matthew Johnson <matthew@anandabits.com <mailto:matthew@anandabits.com>> wrote:

On Mar 4, 2017, at 9:56 AM, Karim Nassar <karim@karimnassar.com <mailto:karim@karimnassar.com>> wrote:

On Mar 3, 2017, at 5:21 PM, Matthew Johnson <matthew@anandabits.com <mailto:matthew@anandabits.com>> wrote:

On Mar 3, 2017, at 9:24 AM, Karim Nassar via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Changes that *are* necessary are:

* Change the spelling of `internal` to `module` (making `module` the new default)
* Introduce a new modifier `internal` to mean "Internal to the current sub-module and its child-sub-modules”

Can you give concrete examples of use cases where a descendent submodule needs access to symbols declared by an ancestor? I gave some thought to this when drafting my proposal and came to the conclusion that this runs against the grain of layering and is likely to be a bad idea in practice. If there are use cases I didn’t consider I am very interested in learning about them.

On further reflection and examination of my notes, I think you’re right, and that the `internal` encapsulation should be purely horizontal. Will adjust to reflect that.

### Making a Sub-module

To create a sub-module within a Module (or sub-module) is simple: The author creates a directory, and places a "sub-module declaration file" within the directory:

```
// __submodule.swift

Why the double underscore prefix? To make it sort to the top in a file browser?

Is this file allowed to have any Swift code? Or is it limited to submodule-related declarations only? If the latter, why not use an extension such as `.submodule` or `.swiftmodule` to differentiate it from ordinary Swift files and allow the submodule to be named by the name of this file?

So, my reasoning was that by requiring a specific standard name for the declaration file, we guarantee that any directory can only describe one submodule. Prefixing the proposed name with underscores was simply a way of preventing naming collision with potential “real code” files (and yes, as a side-effect alpha-floating it to the top). Since the `submodule` declaration might expand to include statements & configuration about the sub-module, I see no reason to prohibit Swift code from existing in that sub-module declaration file… Disallowing/controlling that seems to be a style/linter concern.

However, as I mentioned above, all the specific spellings (except `internal`) for the different concepts in this proposal are straw-men awaiting input. I’d say the addition of a new type of file extension raises some concerns for me, but there’s already been a lot of push back on the general idea of using filesystem structures to organize sub-modules, so the whole idea may be moot.

I’ve actually been starting to come around to the idea of using the file system. Not so much because I really like it, but because I have been considering further some of the drawbacks of other approaches.

One big reason is that a submodule should form a scope and scopes should consist of code that is physically adjacent. In practice this means it should reside in close proximity in the file system. Allowing any file in a project to be a part of any submodule partly defeats the purpose of using them to structure a project internally. If we’re going to be organizing the files in a submodule physically anyway maybe we should just take advantage of that fact and prevent a stray file in a distant part of the file system from being part of the submodule.

The second reason is that there is a big problem with placing a simple `submodule Foo` declaration at the top of each file in a submodule. We all make typos from time to time. This method makes it too easy to accidentally type `submodule Fooo` and end up in a submodule you didn’t intend. This mistake would likely be caught relatively quickly but it seems silly to have a system subject to this kind of mistake. This means we would end arbitrarily placing the declaration in one file and saying `extends submodule Foo` in the rest of the files. Your design avoids the need to arbitrarily choose where to place the declaration and avoids the need for a declaration in the rest of the files.

I’m not dead-set on this approach, but as you say, it solves a *lot* of problems that other approaches introduce. I do recognize the reasonableness of the main argument against, that file location shouldn’t have such a dramatic affect on behavior… *but* the fact is (whether by convention or expediency) we already *do* have filesystem location dependencies on our code projects…

* One can’t willy-nilly move files around the disk and expect a project to build… if you move it somewhere the compiler doesn’t know to look for it, you’re going to break things
* One can’t just move a file out of your SCM repo root, or you’ll “lose” the file

True, but if other files do not refer to the lost file, you don’t even know for which file to look.
(imo this is already a weak point in swift, but this approach to submodules would make it -much?- worse)

If you were to include filenames in the “submodule declaration file” at least this omission in swift would be partly covered, and you would not need to move the other files into special places. (You could though)

But isn’t this exactly the same case if you’re using Swift without an IDE? It’s true that Xcode keeps track of files in your project, and there’s no reason it couldn’t also keep track of files in a submodule (in fact it would… it just might not know they belong in the sub-module without some work from the Xcode dev team).

But as a thought-experiment, here’s how I’m thinking of this problem:

Swift (as a language) has established that a file *is* a unit of code. Not the only one of course, but it is one, as evidenced by access controls (fileprivate, Swift2 private) which bound on the file.

I (and many others) want to be able to describe another unit of code larger than a file and smaller than a Module to help organize the file-units we have which go together within a Module, but which shouldn’t (for various reasons) describe a complete Module in and of themselves.

For the moment—though we actually propose to call this unit a sub-module—we'll call this new unit a Foo. Let's describe properties of a Foo:

* A Foo should serve to organize one or more File code units by:
  - ensuring a File (the code-unit) may belong to one and only one Foo
  - making clear (with or without an IDE) to which Foo a given file might belong
* A Foo should have a unique name
* A Foo should be able to nest under another Foo, and the name of that nesting should be clear & unique
* A Foo should be able to be easily created, split, and merged as the codebase evolves
* A Foo should be able to be moved, copied, and SCMed as a unit, without reference to anything outside the Foo

Having now described the critical properties of this new unit called “Foo” have we not also described a filesystem directory?

Naturally, we don’t want *all* directories to become Foos (that would be overly restrictive to authors, and backwards-incompatible), so we need a way to distinguish a normal directory from one that is acting as a Foo. I originally proposed a special “sub-module declaration file”, but there may be other ways. For example, we might establish that a directory whose name matches `[SubModuleName].swift-sub-module` becomes a sub-module of designated name, however this can cause problems due to the disjoint sets of “Legal Filesystem Directory Characters” and “Legal Swift Identifier Characters”. It also doesn’t give us a place to “grow” the declaration in the future if we choose to add additional properties to the sub-module declaration.

—Karim

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Punycode would readily cover your first issue about legal identifier characters.

Well, it would cover… I’m not sure if “readily” would be the right word.

If I have a sub-module I want to call `Bücher`, and I have to name the directory `Bcher-kva`, what do I use to refer to the sub-module in my code? It’s not at all readable, nor is it discoverable to a 3rd party who if they don’t “know” about Punycode, have no reason to associate the `Bücher` sub-module with the `Bcher-kva` directory.

Not that’s a show-stopper, mind. I wouldn’t mind further exploring the directory-name + “extension" approach.

—Karim

···

On Mar 5, 2017, at 2:06 PM, Xiaodi Wu <xiaodi.wu@gmail.com> wrote:

As to the second, having enumerated the essential properties of a Foo, what "growth" are you envisioning?
On Sun, Mar 5, 2017 at 12:51 Karim Nassar via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Mar 5, 2017, at 10:10 AM, Rien <Rien@Balancingrock.nl <mailto:Rien@Balancingrock.nl>> wrote:

On 05 Mar 2017, at 15:52, Karim Nassar via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

On Mar 4, 2017, at 2:54 PM, Matthew Johnson <matthew@anandabits.com <mailto:matthew@anandabits.com>> wrote:

On Mar 4, 2017, at 9:56 AM, Karim Nassar <karim@karimnassar.com <mailto:karim@karimnassar.com>> wrote:

On Mar 3, 2017, at 5:21 PM, Matthew Johnson <matthew@anandabits.com <mailto:matthew@anandabits.com>> wrote:

On Mar 3, 2017, at 9:24 AM, Karim Nassar via swift-evolution <swift-evolution@swift.org <mailto:swift-evolution@swift.org>> wrote:

Changes that *are* necessary are:

* Change the spelling of `internal` to `module` (making `module` the new default)
* Introduce a new modifier `internal` to mean "Internal to the current sub-module and its child-sub-modules”

Can you give concrete examples of use cases where a descendent submodule needs access to symbols declared by an ancestor? I gave some thought to this when drafting my proposal and came to the conclusion that this runs against the grain of layering and is likely to be a bad idea in practice. If there are use cases I didn’t consider I am very interested in learning about them.

On further reflection and examination of my notes, I think you’re right, and that the `internal` encapsulation should be purely horizontal. Will adjust to reflect that.

### Making a Sub-module

To create a sub-module within a Module (or sub-module) is simple: The author creates a directory, and places a "sub-module declaration file" within the directory:

```
// __submodule.swift

Why the double underscore prefix? To make it sort to the top in a file browser?

Is this file allowed to have any Swift code? Or is it limited to submodule-related declarations only? If the latter, why not use an extension such as `.submodule` or `.swiftmodule` to differentiate it from ordinary Swift files and allow the submodule to be named by the name of this file?

So, my reasoning was that by requiring a specific standard name for the declaration file, we guarantee that any directory can only describe one submodule. Prefixing the proposed name with underscores was simply a way of preventing naming collision with potential “real code” files (and yes, as a side-effect alpha-floating it to the top). Since the `submodule` declaration might expand to include statements & configuration about the sub-module, I see no reason to prohibit Swift code from existing in that sub-module declaration file… Disallowing/controlling that seems to be a style/linter concern.

However, as I mentioned above, all the specific spellings (except `internal`) for the different concepts in this proposal are straw-men awaiting input. I’d say the addition of a new type of file extension raises some concerns for me, but there’s already been a lot of push back on the general idea of using filesystem structures to organize sub-modules, so the whole idea may be moot.

I’ve actually been starting to come around to the idea of using the file system. Not so much because I really like it, but because I have been considering further some of the drawbacks of other approaches.

One big reason is that a submodule should form a scope and scopes should consist of code that is physically adjacent. In practice this means it should reside in close proximity in the file system. Allowing any file in a project to be a part of any submodule partly defeats the purpose of using them to structure a project internally. If we’re going to be organizing the files in a submodule physically anyway maybe we should just take advantage of that fact and prevent a stray file in a distant part of the file system from being part of the submodule.

The second reason is that there is a big problem with placing a simple `submodule Foo` declaration at the top of each file in a submodule. We all make typos from time to time. This method makes it too easy to accidentally type `submodule Fooo` and end up in a submodule you didn’t intend. This mistake would likely be caught relatively quickly but it seems silly to have a system subject to this kind of mistake. This means we would end arbitrarily placing the declaration in one file and saying `extends submodule Foo` in the rest of the files. Your design avoids the need to arbitrarily choose where to place the declaration and avoids the need for a declaration in the rest of the files.

I’m not dead-set on this approach, but as you say, it solves a *lot* of problems that other approaches introduce. I do recognize the reasonableness of the main argument against, that file location shouldn’t have such a dramatic affect on behavior… *but* the fact is (whether by convention or expediency) we already *do* have filesystem location dependencies on our code projects…

* One can’t willy-nilly move files around the disk and expect a project to build… if you move it somewhere the compiler doesn’t know to look for it, you’re going to break things
* One can’t just move a file out of your SCM repo root, or you’ll “lose” the file

True, but if other files do not refer to the lost file, you don’t even know for which file to look.
(imo this is already a weak point in swift, but this approach to submodules would make it -much?- worse)

If you were to include filenames in the “submodule declaration file” at least this omission in swift would be partly covered, and you would not need to move the other files into special places. (You could though)

But isn’t this exactly the same case if you’re using Swift without an IDE? It’s true that Xcode keeps track of files in your project, and there’s no reason it couldn’t also keep track of files in a submodule (in fact it would… it just might not know they belong in the sub-module without some work from the Xcode dev team).

But as a thought-experiment, here’s how I’m thinking of this problem:

Swift (as a language) has established that a file *is* a unit of code. Not the only one of course, but it is one, as evidenced by access controls (fileprivate, Swift2 private) which bound on the file.

I (and many others) want to be able to describe another unit of code larger than a file and smaller than a Module to help organize the file-units we have which go together within a Module, but which shouldn’t (for various reasons) describe a complete Module in and of themselves.

For the moment—though we actually propose to call this unit a sub-module—we'll call this new unit a Foo. Let's describe properties of a Foo:

* A Foo should serve to organize one or more File code units by:
  - ensuring a File (the code-unit) may belong to one and only one Foo
  - making clear (with or without an IDE) to which Foo a given file might belong
* A Foo should have a unique name
* A Foo should be able to nest under another Foo, and the name of that nesting should be clear & unique
* A Foo should be able to be easily created, split, and merged as the codebase evolves
* A Foo should be able to be moved, copied, and SCMed as a unit, without reference to anything outside the Foo

Having now described the critical properties of this new unit called “Foo” have we not also described a filesystem directory?

Naturally, we don’t want *all* directories to become Foos (that would be overly restrictive to authors, and backwards-incompatible), so we need a way to distinguish a normal directory from one that is acting as a Foo. I originally proposed a special “sub-module declaration file”, but there may be other ways. For example, we might establish that a directory whose name matches `[SubModuleName].swift-sub-module` becomes a sub-module of designated name, however this can cause problems due to the disjoint sets of “Legal Filesystem Directory Characters” and “Legal Swift Identifier Characters”. It also doesn’t give us a place to “grow” the declaration in the future if we choose to add additional properties to the sub-module declaration.

—Karim

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org <mailto:swift-evolution@swift.org>
https://lists.swift.org/mailman/listinfo/swift-evolution

Left unsaid from my earlier message was that I wonder why an explicit
submodule declaration would be necessary at all if you commit to the
principles laid out by Karim.

···

On Sun, Mar 5, 2017 at 13:33 Matthew Johnson <matthew@anandabits.com> wrote:

On Mar 5, 2017, at 1:29 PM, Karim Nassar via swift-evolution < > swift-evolution@swift.org> wrote:

On Mar 5, 2017, at 2:06 PM, Xiaodi Wu <xiaodi.wu@gmail.com> wrote:

Punycode would readily cover your first issue about legal identifier
characters.

Well, it would cover… I’m not sure if “readily” would be the right word.

If I have a sub-module I want to call `Bücher`, and I have to name the
directory `Bcher-kva`, what do I use to refer to the sub-module in my code?
It’s not at all readable, nor is it discoverable to a 3rd party who if they
don’t “know” about Punycode, have no reason to associate the `Bücher`
sub-module with the `Bcher-kva` directory.

Not that’s a show-stopper, mind. I wouldn’t mind further exploring the
directory-name + “extension" approach.

If we explore this I think it needs to be as a form of “syntactic sugar”
only. We still need a place to write an explicit submodule declaration.

—Karim

As to the second, having enumerated the essential properties of a Foo,
what "growth" are you envisioning?
On Sun, Mar 5, 2017 at 12:51 Karim Nassar via swift-evolution < > swift-evolution@swift.org> wrote:

On Mar 5, 2017, at 10:10 AM, Rien <Rien@Balancingrock.nl> wrote:

On 05 Mar 2017, at 15:52, Karim Nassar via swift-evolution < > swift-evolution@swift.org> wrote:

On Mar 4, 2017, at 2:54 PM, Matthew Johnson <matthew@anandabits.com> > wrote:

On Mar 4, 2017, at 9:56 AM, Karim Nassar <karim@karimnassar.com> wrote:

On Mar 3, 2017, at 5:21 PM, Matthew Johnson <matthew@anandabits.com> > wrote:

On Mar 3, 2017, at 9:24 AM, Karim Nassar via swift-evolution < > swift-evolution@swift.org> wrote:

Changes that *are* necessary are:

* Change the spelling of `internal` to `module` (making `module` the new
default)
* Introduce a new modifier `internal` to mean "Internal to the current
sub-module and its child-sub-modules”

Can you give concrete examples of use cases where a descendent submodule
needs access to symbols declared by an ancestor? I gave some thought to
this when drafting my proposal and came to the conclusion that this runs
against the grain of layering and is likely to be a bad idea in practice.
If there are use cases I didn’t consider I am very interested in learning
about them.

On further reflection and examination of my notes, I think you’re right,
and that the `internal` encapsulation should be purely horizontal. Will
adjust to reflect that.

### Making a Sub-module

To create a sub-module within a Module (or sub-module) is simple: The
author creates a directory, and places a "sub-module declaration file"
within the directory:

```
// __submodule.swift

Why the double underscore prefix? To make it sort to the top in a file
browser?

Is this file allowed to have any Swift code? Or is it limited to
submodule-related declarations only? If the latter, why not use an
extension such as `.submodule` or `.swiftmodule` to differentiate it from
ordinary Swift files and allow the submodule to be named by the name of
this file?

So, my reasoning was that by requiring a specific standard name for the
declaration file, we guarantee that any directory can only describe one
submodule. Prefixing the proposed name with underscores was simply a way of
preventing naming collision with potential “real code” files (and yes, as a
side-effect alpha-floating it to the top). Since the `submodule`
declaration might expand to include statements & configuration about the
sub-module, I see no reason to prohibit Swift code from existing in that
sub-module declaration file… Disallowing/controlling that seems to be a
style/linter concern.

However, as I mentioned above, all the specific spellings (except
`internal`) for the different concepts in this proposal are straw-men
awaiting input. I’d say the addition of a new type of file extension raises
some concerns for me, but there’s already been a lot of push back on the
general idea of using filesystem structures to organize sub-modules, so the
whole idea may be moot.

I’ve actually been starting to come around to the idea of using the file
system. Not so much because I really like it, but because I have been
considering further some of the drawbacks of other approaches.

One big reason is that a submodule should form a scope and scopes should
consist of code that is physically adjacent. In practice this means it
should reside in close proximity in the file system. Allowing any file in
a project to be a part of any submodule partly defeats the purpose of using
them to structure a project internally. If we’re going to be organizing
the files in a submodule physically anyway maybe we should just take
advantage of that fact and prevent a stray file in a distant part of the
file system from being part of the submodule.

The second reason is that there is a big problem with placing a simple
`submodule Foo` declaration at the top of each file in a submodule. We all
make typos from time to time. This method makes it too easy to
accidentally type `submodule Fooo` and end up in a submodule you didn’t
intend. This mistake would likely be caught relatively quickly but it
seems silly to have a system subject to this kind of mistake. This means
we would end arbitrarily placing the declaration in one file and saying
`extends submodule Foo` in the rest of the files. Your design avoids the
need to arbitrarily choose where to place the declaration and avoids the
need for a declaration in the rest of the files.

I’m not dead-set on this approach, but as you say, it solves a *lot* of
problems that other approaches introduce. I do recognize the reasonableness
of the main argument against, that file location shouldn’t have such a
dramatic affect on behavior… *but* the fact is (whether by convention or
expediency) we already *do* have filesystem location dependencies on our
code projects…

* One can’t willy-nilly move files around the disk and expect a project to
build… if you move it somewhere the compiler doesn’t know to look for it,
you’re going to break things
* One can’t just move a file out of your SCM repo root, or you’ll “lose”
the file

True, but if other files do not refer to the lost file, you don’t even
know for which file to look.
(imo this is already a weak point in swift, but this approach to
submodules would make it -much?- worse)

If you were to include filenames in the “submodule declaration file” at
least this omission in swift would be partly covered, and you would not
need to move the other files into special places. (You could though)

But isn’t this exactly the same case if you’re using Swift without an IDE?
It’s true that Xcode keeps track of files in your project, and there’s no
reason it couldn’t also keep track of files in a submodule (in fact it
would… it just might not know they belong in the sub-module without some
work from the Xcode dev team).

But as a thought-experiment, here’s how I’m thinking of this problem:

Swift (as a language) has established that a file *is* a unit of code. Not
the only one of course, but it is one, as evidenced by access controls
(fileprivate, Swift2 private) which bound on the file.

I (and many others) want to be able to describe another unit of code
larger than a file and smaller than a Module to help organize the
file-units we have which go together within a Module, but which shouldn’t
(for various reasons) describe a complete Module in and of themselves.

For the moment—though we actually propose to call this unit a
sub-module—we'll call this new unit a Foo. Let's describe properties of a
Foo:

* A Foo should serve to organize one or more File code units by:
  - ensuring a File (the code-unit) may belong to one and only one Foo
  - making clear (with or without an IDE) to which Foo a given file might
belong
* A Foo should have a unique name
* A Foo should be able to nest under another Foo, and the name of that
nesting should be clear & unique
* A Foo should be able to be easily created, split, and merged as the
codebase evolves
* A Foo should be able to be moved, copied, and SCMed as a unit, without
reference to anything outside the Foo

Having now described the critical properties of this new unit called “Foo”
have we not also described a filesystem directory?

Naturally, we don’t want *all* directories to become Foos (that would be
overly restrictive to authors, and backwards-incompatible), so we need a
way to distinguish a normal directory from one that is acting as a Foo. I
originally proposed a special “sub-module declaration file”, but there may
be other ways. For example, we might establish that a directory whose name
matches `[SubModuleName].swift-sub-module` becomes a sub-module of
designated name, however this can cause problems due to the disjoint sets
of “Legal Filesystem Directory Characters” and “Legal Swift Identifier
Characters”. It also doesn’t give us a place to “grow” the declaration in
the future if we choose to add additional properties to the sub-module
declaration.

—Karim

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

All modern file systems have good Unicode support--even FAT32 with LFN. Do
you know of any that stop you from naming a file "Bücher.swift"? Punycode
would be a possible fallback on the rare occasion you'd need to exceed the
capacities of the file system, but I don't imagine this would happen often
if at all.

···

On Sun, Mar 5, 2017 at 13:29 Karim Nassar <karim@karimnassar.com> wrote:

On Mar 5, 2017, at 2:06 PM, Xiaodi Wu <xiaodi.wu@gmail.com> wrote:

Punycode would readily cover your first issue about legal identifier
characters.

Well, it would cover… I’m not sure if “readily” would be the right word.

If I have a sub-module I want to call `Bücher`, and I have to name the
directory `Bcher-kva`, what do I use to refer to the sub-module in my code?
It’s not at all readable, nor is it discoverable to a 3rd party who if they
don’t “know” about Punycode, have no reason to associate the `Bücher`
sub-module with the `Bcher-kva` directory.

Not that’s a show-stopper, mind. I wouldn’t mind further exploring the
directory-name + “extension" approach.

—Karim

As to the second, having enumerated the essential properties of a Foo,
what "growth" are you envisioning?
On Sun, Mar 5, 2017 at 12:51 Karim Nassar via swift-evolution < > swift-evolution@swift.org> wrote:

On Mar 5, 2017, at 10:10 AM, Rien <Rien@Balancingrock.nl> wrote:

On 05 Mar 2017, at 15:52, Karim Nassar via swift-evolution < > swift-evolution@swift.org> wrote:

On Mar 4, 2017, at 2:54 PM, Matthew Johnson <matthew@anandabits.com> > wrote:

On Mar 4, 2017, at 9:56 AM, Karim Nassar <karim@karimnassar.com> wrote:

On Mar 3, 2017, at 5:21 PM, Matthew Johnson <matthew@anandabits.com> > wrote:

On Mar 3, 2017, at 9:24 AM, Karim Nassar via swift-evolution < > swift-evolution@swift.org> wrote:

Changes that *are* necessary are:

* Change the spelling of `internal` to `module` (making `module` the new
default)
* Introduce a new modifier `internal` to mean "Internal to the current
sub-module and its child-sub-modules”

Can you give concrete examples of use cases where a descendent submodule
needs access to symbols declared by an ancestor? I gave some thought to
this when drafting my proposal and came to the conclusion that this runs
against the grain of layering and is likely to be a bad idea in practice.
If there are use cases I didn’t consider I am very interested in learning
about them.

On further reflection and examination of my notes, I think you’re right,
and that the `internal` encapsulation should be purely horizontal. Will
adjust to reflect that.

### Making a Sub-module

To create a sub-module within a Module (or sub-module) is simple: The
author creates a directory, and places a "sub-module declaration file"
within the directory:

```
// __submodule.swift

Why the double underscore prefix? To make it sort to the top in a file
browser?

Is this file allowed to have any Swift code? Or is it limited to
submodule-related declarations only? If the latter, why not use an
extension such as `.submodule` or `.swiftmodule` to differentiate it from
ordinary Swift files and allow the submodule to be named by the name of
this file?

So, my reasoning was that by requiring a specific standard name for the
declaration file, we guarantee that any directory can only describe one
submodule. Prefixing the proposed name with underscores was simply a way of
preventing naming collision with potential “real code” files (and yes, as a
side-effect alpha-floating it to the top). Since the `submodule`
declaration might expand to include statements & configuration about the
sub-module, I see no reason to prohibit Swift code from existing in that
sub-module declaration file… Disallowing/controlling that seems to be a
style/linter concern.

However, as I mentioned above, all the specific spellings (except
`internal`) for the different concepts in this proposal are straw-men
awaiting input. I’d say the addition of a new type of file extension raises
some concerns for me, but there’s already been a lot of push back on the
general idea of using filesystem structures to organize sub-modules, so the
whole idea may be moot.

I’ve actually been starting to come around to the idea of using the file
system. Not so much because I really like it, but because I have been
considering further some of the drawbacks of other approaches.

One big reason is that a submodule should form a scope and scopes should
consist of code that is physically adjacent. In practice this means it
should reside in close proximity in the file system. Allowing any file in
a project to be a part of any submodule partly defeats the purpose of using
them to structure a project internally. If we’re going to be organizing
the files in a submodule physically anyway maybe we should just take
advantage of that fact and prevent a stray file in a distant part of the
file system from being part of the submodule.

The second reason is that there is a big problem with placing a simple
`submodule Foo` declaration at the top of each file in a submodule. We all
make typos from time to time. This method makes it too easy to
accidentally type `submodule Fooo` and end up in a submodule you didn’t
intend. This mistake would likely be caught relatively quickly but it
seems silly to have a system subject to this kind of mistake. This means
we would end arbitrarily placing the declaration in one file and saying
`extends submodule Foo` in the rest of the files. Your design avoids the
need to arbitrarily choose where to place the declaration and avoids the
need for a declaration in the rest of the files.

I’m not dead-set on this approach, but as you say, it solves a *lot* of
problems that other approaches introduce. I do recognize the reasonableness
of the main argument against, that file location shouldn’t have such a
dramatic affect on behavior… *but* the fact is (whether by convention or
expediency) we already *do* have filesystem location dependencies on our
code projects…

* One can’t willy-nilly move files around the disk and expect a project to
build… if you move it somewhere the compiler doesn’t know to look for it,
you’re going to break things
* One can’t just move a file out of your SCM repo root, or you’ll “lose”
the file

True, but if other files do not refer to the lost file, you don’t even
know for which file to look.
(imo this is already a weak point in swift, but this approach to
submodules would make it -much?- worse)

If you were to include filenames in the “submodule declaration file” at
least this omission in swift would be partly covered, and you would not
need to move the other files into special places. (You could though)

But isn’t this exactly the same case if you’re using Swift without an IDE?
It’s true that Xcode keeps track of files in your project, and there’s no
reason it couldn’t also keep track of files in a submodule (in fact it
would… it just might not know they belong in the sub-module without some
work from the Xcode dev team).

But as a thought-experiment, here’s how I’m thinking of this problem:

Swift (as a language) has established that a file *is* a unit of code. Not
the only one of course, but it is one, as evidenced by access controls
(fileprivate, Swift2 private) which bound on the file.

I (and many others) want to be able to describe another unit of code
larger than a file and smaller than a Module to help organize the
file-units we have which go together within a Module, but which shouldn’t
(for various reasons) describe a complete Module in and of themselves.

For the moment—though we actually propose to call this unit a
sub-module—we'll call this new unit a Foo. Let's describe properties of a
Foo:

* A Foo should serve to organize one or more File code units by:
  - ensuring a File (the code-unit) may belong to one and only one Foo
  - making clear (with or without an IDE) to which Foo a given file might
belong
* A Foo should have a unique name
* A Foo should be able to nest under another Foo, and the name of that
nesting should be clear & unique
* A Foo should be able to be easily created, split, and merged as the
codebase evolves
* A Foo should be able to be moved, copied, and SCMed as a unit, without
reference to anything outside the Foo

Having now described the critical properties of this new unit called “Foo”
have we not also described a filesystem directory?

Naturally, we don’t want *all* directories to become Foos (that would be
overly restrictive to authors, and backwards-incompatible), so we need a
way to distinguish a normal directory from one that is acting as a Foo. I
originally proposed a special “sub-module declaration file”, but there may
be other ways. For example, we might establish that a directory whose name
matches `[SubModuleName].swift-sub-module` becomes a sub-module of
designated name, however this can cause problems due to the disjoint sets
of “Legal Filesystem Directory Characters” and “Legal Swift Identifier
Characters”. It also doesn’t give us a place to “grow” the declaration in
the future if we choose to add additional properties to the sub-module
declaration.

—Karim

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

I’m not dead-set on this approach, but as you say, it solves a *lot* of problems that other approaches introduce. I do recognize the reasonableness of the main argument against, that file location shouldn’t have such a dramatic affect on behavior… *but* the fact is (whether by convention or expediency) we already *do* have filesystem location dependencies on our code projects…

* One can’t willy-nilly move files around the disk and expect a project to build… if you move it somewhere the compiler doesn’t know to look for it, you’re going to break things
* One can’t just move a file out of your SCM repo root, or you’ll “lose” the file

True, but if other files do not refer to the lost file, you don’t even know for which file to look.
(imo this is already a weak point in swift, but this approach to submodules would make it -much?- worse)

If you were to include filenames in the “submodule declaration file” at least this omission in swift would be partly covered, and you would not need to move the other files into special places. (You could though)

But isn’t this exactly the same case if you’re using Swift without an IDE? It’s true that Xcode keeps track of files in your project, and there’s no reason it couldn’t also keep track of files in a submodule (in fact it would… it just might not know they belong in the sub-module without some work from the Xcode dev team).

But as a thought-experiment, here’s how I’m thinking of this problem:

Swift (as a language) has established that a file *is* a unit of code. Not the only one of course, but it is one, as evidenced by access controls (fileprivate, Swift2 private) which bound on the file.

I disagree with this: Swift is just a set of -more or less- arbitrary rules to allow the development of applications.
The criteria for those rules are chosen by the core team and will fluctuate with time.
As such there is no “Swift way” of doing things. Any such remark is by its nature a personal preference. (not that there is anything wrong with that, its just not an argument)

(Besides, a computer only knows files, thus of course a file is a unit of code. That does not mean we should extend “code” to be something outside the files)

First, that’s simply not true. In Swift 1, a file was *not* a logical unit of code because file boundaries were meaningless to the way the code in the project behaved. In Swift 2 we got `private` which made it possible for file boundaries to affect code behavior. Swift 3 changed `private` but introduced `fileprivate` for the same purpose.

Second, yes, Swift is just a language, but as that language evolves, it is an expression of the values & preferences of its development team and more recently, this community. If the file was not *intended* to be a unit of code, we would not have `fileprivate`.

I (and many others) want to be able to describe another unit of code larger than a file and smaller than a Module to help organize the file-units we have which go together within a Module, but which shouldn’t (for various reasons) describe a complete Module in and of themselves.

Sure, and I have nothing against that. However I an dead set against extending the scope of the language itself to the organisation of its files. It is traditional to leave the organisation of the files to the configuration of the compiler/linker. And I do not see how it would simplify the language itself by extending the language to cover file organisation.

Clearly adding any new feature will not simplify the language. The questions are 1) should we add this feature, and 2) how to add the feature. I *do* take your objections to heart and that’s why I added some alternatives not involving the filesystem to my proposal. These alternatives each have their own problems and complications which we can discuss… maybe they’re not as bad as they seem, maybe they’re worse, and *probably*—because I’m just one person—there are solutions to the problem which I haven’t thought of.

I will predict that if this approach is taken you should brace for the storm that would want to reverse this feature. It would imo dwarf the private vs fileprivate debate.

And no one wants that. That’s why we are having this discussion, no? I’ll be honest, I saw the fileprivate/private discussion on this list, and didn’t voice my opinion then because I thought “oh, they’ll do the right thing”, then was unhappy with what we got. That’s the reason I’ve started this discussion, and it’s also the reason I *really* value your dissenting input. I want not just *a* solution, but *the right* solution, and I know I don’t know everything that needs knowing to come up with it alone.

I and many others want a unit between Module & File. Someone else may not want such a thing, but they don’t have to use it, just like they don’t actually have to use Access Control modifiers. Now, you might object, saying *no one* should be able to organize their code this way… Then I’ll disagree with you on that, and we can discuss that position (and again, maybe I’m wrong, but my experience says otherwise). If you accept that wanting that in-between unit is a reasonable thing for us to want (even if you don’t plan to use it), then lets discuss the second question: “How to do it”.

One of my goals with this is that it should add *zero* additional ceremony should an author choose not to use it, and have no impact on existing code.

Besides, why create something new? (Yes there are systems out there that do some of this) There are many existing solutions out there that we could adopt and stay completely within the bounds of the language, and give the users something they are likely already familiar with.

I’ve laid out some alternatives, and *to me* the filesystem-based model *currently* looks the best (or at least, “least bad”). You disagree, and that’s fine. And as I’ve said more than once, I’m open to suggestions.

Many existing solutions that I’ve looked at from other languages face complications when overlaid on Swift due to Swift’s extensions, and the exact structural topology we’re working within (the intersection of: Access Control, type declarations, profligate type extensions, lack of headers, etc), means that a solution for the in-between unit we’re calling ‘sub-modules’ (for lack of a definitive name) *might* need to be somewhat unique to Swift. Or might not.

We also all want to do what is “right” for the Swift language. Many people were calling for a `protected` access control modifier. That it was common in other languages didn't mean it was right for Swift.

At the end of the day, I don’t know what the right solution is. This “problem” came about due to the meaning of `private` (and then `fileprivate`), and the “simple” approach taken to access control, which caused us to organize code around the file, causing some *very* large files. Some time back, many people proposed solving this problem through additional Access Control modifiers to "fill in the gaps” between `internal` and `fileprivate`. There has been a lot of discussion on that score, and time and again, the consensus seemed to be “This smells like a sub-modules problem, lets solve it there”. Very well, lets. The problem hasn’t gone away (for those of us who have it).

For the moment—though we actually propose to call this unit a sub-module—we'll call this new unit a Foo. Let's describe properties of a Foo:

* A Foo should serve to organize one or more File code units by:
- ensuring a File (the code-unit) may belong to one and only one Foo
- making clear (with or without an IDE) to which Foo a given file might belong
* A Foo should have a unique name
* A Foo should be able to nest under another Foo, and the name of that nesting should be clear & unique
* A Foo should be able to be easily created, split, and merged as the codebase evolves
* A Foo should be able to be moved, copied, and SCMed as a unit, without reference to anything outside the Foo

Having now described the critical properties of this new unit called “Foo” have we not also described a filesystem directory?

Many things in life share characteristics. And while we should certainly look to them for lessons, we should not conclude that they are the same.

And that’s also fair. As I said, it was a thought-experiment trying to explain how I got to the directories concept. If it’s wrong, then it’s wrong.

—Karim

···

On Mar 6, 2017, at 3:04 AM, Rien <Rien@Balancingrock.nl> wrote:

Well, my thought-experiment was a way of explaining the essential characteristics of a sub-module... not intended as an exhaustive enumeration of all possible characteristics.

--Karim

···

On Mar 5, 2017, at 3:07 PM, Xiaodi Wu <xiaodi.wu@gmail.com> wrote:

Left unsaid from my earlier message was that I wonder why an explicit submodule declaration would be necessary at all if you commit to the principles laid out by Karim.
On Sun, Mar 5, 2017 at 13:33 Matthew Johnson <matthew@anandabits.com> wrote:

On Mar 5, 2017, at 1:29 PM, Karim Nassar via swift-evolution <swift-evolution@swift.org> wrote:

On Mar 5, 2017, at 2:06 PM, Xiaodi Wu <xiaodi.wu@gmail.com> wrote:

Punycode would readily cover your first issue about legal identifier characters.

Well, it would cover… I’m not sure if “readily” would be the right word.

If I have a sub-module I want to call `Bücher`, and I have to name the directory `Bcher-kva`, what do I use to refer to the sub-module in my code? It’s not at all readable, nor is it discoverable to a 3rd party who if they don’t “know” about Punycode, have no reason to associate the `Bücher` sub-module with the `Bcher-kva` directory.

Not that’s a show-stopper, mind. I wouldn’t mind further exploring the directory-name + “extension" approach.

If we explore this I think it needs to be as a form of “syntactic sugar” only. We still need a place to write an explicit submodule declaration.

—Karim

As to the second, having enumerated the essential properties of a Foo, what "growth" are you envisioning?

On Sun, Mar 5, 2017 at 12:51 Karim Nassar via swift-evolution <swift-evolution@swift.org> wrote:

On Mar 5, 2017, at 10:10 AM, Rien <Rien@Balancingrock.nl> wrote:

On 05 Mar 2017, at 15:52, Karim Nassar via swift-evolution <swift-evolution@swift.org> wrote:

On Mar 4, 2017, at 2:54 PM, Matthew Johnson <matthew@anandabits.com> wrote:

On Mar 4, 2017, at 9:56 AM, Karim Nassar <karim@karimnassar.com> wrote:

On Mar 3, 2017, at 5:21 PM, Matthew Johnson <matthew@anandabits.com> wrote:

On Mar 3, 2017, at 9:24 AM, Karim Nassar via swift-evolution <swift-evolution@swift.org> wrote:

Changes that *are* necessary are:

* Change the spelling of `internal` to `module` (making `module` the new default)
* Introduce a new modifier `internal` to mean "Internal to the current sub-module and its child-sub-modules”

Can you give concrete examples of use cases where a descendent submodule needs access to symbols declared by an ancestor? I gave some thought to this when drafting my proposal and came to the conclusion that this runs against the grain of layering and is likely to be a bad idea in practice. If there are use cases I didn’t consider I am very interested in learning about them.

On further reflection and examination of my notes, I think you’re right, and that the `internal` encapsulation should be purely horizontal. Will adjust to reflect that.

### Making a Sub-module

To create a sub-module within a Module (or sub-module) is simple: The author creates a directory, and places a "sub-module declaration file" within the directory:

```
// __submodule.swift

Why the double underscore prefix? To make it sort to the top in a file browser?

Is this file allowed to have any Swift code? Or is it limited to submodule-related declarations only? If the latter, why not use an extension such as `.submodule` or `.swiftmodule` to differentiate it from ordinary Swift files and allow the submodule to be named by the name of this file?

So, my reasoning was that by requiring a specific standard name for the declaration file, we guarantee that any directory can only describe one submodule. Prefixing the proposed name with underscores was simply a way of preventing naming collision with potential “real code” files (and yes, as a side-effect alpha-floating it to the top). Since the `submodule` declaration might expand to include statements & configuration about the sub-module, I see no reason to prohibit Swift code from existing in that sub-module declaration file… Disallowing/controlling that seems to be a style/linter concern.

However, as I mentioned above, all the specific spellings (except `internal`) for the different concepts in this proposal are straw-men awaiting input. I’d say the addition of a new type of file extension raises some concerns for me, but there’s already been a lot of push back on the general idea of using filesystem structures to organize sub-modules, so the whole idea may be moot.

I’ve actually been starting to come around to the idea of using the file system. Not so much because I really like it, but because I have been considering further some of the drawbacks of other approaches.

One big reason is that a submodule should form a scope and scopes should consist of code that is physically adjacent. In practice this means it should reside in close proximity in the file system. Allowing any file in a project to be a part of any submodule partly defeats the purpose of using them to structure a project internally. If we’re going to be organizing the files in a submodule physically anyway maybe we should just take advantage of that fact and prevent a stray file in a distant part of the file system from being part of the submodule.

The second reason is that there is a big problem with placing a simple `submodule Foo` declaration at the top of each file in a submodule. We all make typos from time to time. This method makes it too easy to accidentally type `submodule Fooo` and end up in a submodule you didn’t intend. This mistake would likely be caught relatively quickly but it seems silly to have a system subject to this kind of mistake. This means we would end arbitrarily placing the declaration in one file and saying `extends submodule Foo` in the rest of the files. Your design avoids the need to arbitrarily choose where to place the declaration and avoids the need for a declaration in the rest of the files.

I’m not dead-set on this approach, but as you say, it solves a *lot* of problems that other approaches introduce. I do recognize the reasonableness of the main argument against, that file location shouldn’t have such a dramatic affect on behavior… *but* the fact is (whether by convention or expediency) we already *do* have filesystem location dependencies on our code projects…

* One can’t willy-nilly move files around the disk and expect a project to build… if you move it somewhere the compiler doesn’t know to look for it, you’re going to break things
* One can’t just move a file out of your SCM repo root, or you’ll “lose” the file

True, but if other files do not refer to the lost file, you don’t even know for which file to look.
(imo this is already a weak point in swift, but this approach to submodules would make it -much?- worse)

If you were to include filenames in the “submodule declaration file” at least this omission in swift would be partly covered, and you would not need to move the other files into special places. (You could though)

But isn’t this exactly the same case if you’re using Swift without an IDE? It’s true that Xcode keeps track of files in your project, and there’s no reason it couldn’t also keep track of files in a submodule (in fact it would… it just might not know they belong in the sub-module without some work from the Xcode dev team).

But as a thought-experiment, here’s how I’m thinking of this problem:

Swift (as a language) has established that a file *is* a unit of code. Not the only one of course, but it is one, as evidenced by access controls (fileprivate, Swift2 private) which bound on the file.

I (and many others) want to be able to describe another unit of code larger than a file and smaller than a Module to help organize the file-units we have which go together within a Module, but which shouldn’t (for various reasons) describe a complete Module in and of themselves.

For the moment—though we actually propose to call this unit a sub-module—we'll call this new unit a Foo. Let's describe properties of a Foo:

* A Foo should serve to organize one or more File code units by:
  - ensuring a File (the code-unit) may belong to one and only one Foo
  - making clear (with or without an IDE) to which Foo a given file might belong
* A Foo should have a unique name
* A Foo should be able to nest under another Foo, and the name of that nesting should be clear & unique
* A Foo should be able to be easily created, split, and merged as the codebase evolves
* A Foo should be able to be moved, copied, and SCMed as a unit, without reference to anything outside the Foo

Having now described the critical properties of this new unit called “Foo” have we not also described a filesystem directory?

Naturally, we don’t want *all* directories to become Foos (that would be overly restrictive to authors, and backwards-incompatible), so we need a way to distinguish a normal directory from one that is acting as a Foo. I originally proposed a special “sub-module declaration file”, but there may be other ways. For example, we might establish that a directory whose name matches `[SubModuleName].swift-sub-module` becomes a sub-module of designated name, however this can cause problems due to the disjoint sets of “Legal Filesystem Directory Characters” and “Legal Swift Identifier Characters”. It also doesn’t give us a place to “grow” the declaration in the future if we choose to add additional properties to the sub-module declaration.

—Karim

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

_______________________________________________
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

I’m not dead-set on this approach, but as you say, it solves a *lot* of problems that other approaches introduce. I do recognize the reasonableness of the main argument against, that file location shouldn’t have such a dramatic affect on behavior… *but* the fact is (whether by convention or expediency) we already *do* have filesystem location dependencies on our code projects…

* One can’t willy-nilly move files around the disk and expect a project to build… if you move it somewhere the compiler doesn’t know to look for it, you’re going to break things
* One can’t just move a file out of your SCM repo root, or you’ll “lose” the file

True, but if other files do not refer to the lost file, you don’t even know for which file to look.
(imo this is already a weak point in swift, but this approach to submodules would make it -much?- worse)

If you were to include filenames in the “submodule declaration file” at least this omission in swift would be partly covered, and you would not need to move the other files into special places. (You could though)

But isn’t this exactly the same case if you’re using Swift without an IDE? It’s true that Xcode keeps track of files in your project, and there’s no reason it couldn’t also keep track of files in a submodule (in fact it would… it just might not know they belong in the sub-module without some work from the Xcode dev team).

But as a thought-experiment, here’s how I’m thinking of this problem:

Swift (as a language) has established that a file *is* a unit of code. Not the only one of course, but it is one, as evidenced by access controls (fileprivate, Swift2 private) which bound on the file.

I disagree with this: Swift is just a set of -more or less- arbitrary rules to allow the development of applications.
The criteria for those rules are chosen by the core team and will fluctuate with time.
As such there is no “Swift way” of doing things. Any such remark is by its nature a personal preference. (not that there is anything wrong with that, its just not an argument)

(Besides, a computer only knows files, thus of course a file is a unit of code. That does not mean we should extend “code” to be something outside the files)

First, that’s simply not true. In Swift 1, a file was *not* a logical unit of code because file boundaries were meaningless to the way the code in the project behaved. In Swift 2 we got `private` which made it possible for file boundaries to affect code behavior. Swift 3 changed `private` but introduced `fileprivate` for the same purpose.

Second, yes, Swift is just a language, but as that language evolves, it is an expression of the values & preferences of its development team and more recently, this community. If the file was not *intended* to be a unit of code, we would not have `fileprivate`.

Take this not too personally, but this is a clear example of “cherrypicking”.

For example: Quite clearly swift currently defines modules as an end product of the compile/link process. So if we want multiple modules in a project, we can simply do it that way.

Yet you are clearly willing to throw this rule under the bus and replace it with something new. But if we should not adhere to the rule set in swift, then it is equally not possible to find justification in the current rules.

That is not to say that we should ignore the current rules, because it reduced the cognitive load of developers if new rules closely track exiting rules. But each change (proposal) should be examined against the current rules, and if it does not violate them, or if it does not introduce something that creates cognitive dissonance, then the proposal -at least- should not be rejected because of that.

I.e. when proposing a change, the current rule set -not directly affected by the change- can be used as a reason to reject the proposal, but never to accept it.

I (and many others) want to be able to describe another unit of code larger than a file and smaller than a Module to help organize the file-units we have which go together within a Module, but which shouldn’t (for various reasons) describe a complete Module in and of themselves.

Sure, and I have nothing against that. However I an dead set against extending the scope of the language itself to the organisation of its files. It is traditional to leave the organisation of the files to the configuration of the compiler/linker. And I do not see how it would simplify the language itself by extending the language to cover file organisation.

Clearly adding any new feature will not simplify the language. The questions are 1) should we add this feature, and 2) how to add the feature. I *do* take your objections to heart and that’s why I added some alternatives not involving the filesystem to my proposal. These alternatives each have their own problems and complications which we can discuss… maybe they’re not as bad as they seem, maybe they’re worse, and *probably*—because I’m just one person—there are solutions to the problem which I haven’t thought of.

I will predict that if this approach is taken you should brace for the storm that would want to reverse this feature. It would imo dwarf the private vs fileprivate debate.

And no one wants that. That’s why we are having this discussion, no? I’ll be honest, I saw the fileprivate/private discussion on this list, and didn’t voice my opinion then because I thought “oh, they’ll do the right thing”, then was unhappy with what we got. That’s the reason I’ve started this discussion, and it’s also the reason I *really* value your dissenting input. I want not just *a* solution, but *the right* solution, and I know I don’t know everything that needs knowing to come up with it alone.

I and many others want a unit between Module & File. Someone else may not want such a thing, but they don’t have to use it, just like they don’t actually have to use Access Control modifiers. Now, you might object, saying *no one* should be able to organize their code this way… Then I’ll disagree with you on that, and we can discuss that position (and again, maybe I’m wrong, but my experience says otherwise). If you accept that wanting that in-between unit is a reasonable thing for us to want (even if you don’t plan to use it), then lets discuss the second question: “How to do it”.

One of my goals with this is that it should add *zero* additional ceremony should an author choose not to use it, and have no impact on existing code.

Besides, why create something new? (Yes there are systems out there that do some of this) There are many existing solutions out there that we could adopt and stay completely within the bounds of the language, and give the users something they are likely already familiar with.

I’ve laid out some alternatives, and *to me* the filesystem-based model *currently* looks the best (or at least, “least bad”). You disagree, and that’s fine. And as I’ve said more than once, I’m open to suggestions.

I’d be interested in your opinion on my counter proposal that I posted yesterday. Is there anything in there that would create a problem for you?

Many existing solutions that I’ve looked at from other languages face complications when overlaid on Swift due to Swift’s extensions, and the exact structural topology we’re working within (the intersection of: Access Control, type declarations, profligate type extensions, lack of headers, etc), means that a solution for the in-between unit we’re calling ‘sub-modules’ (for lack of a definitive name) *might* need to be somewhat unique to Swift. Or might not.

We also all want to do what is “right” for the Swift language. Many people were calling for a `protected` access control modifier. That it was common in other languages didn't mean it was right for Swift.

At the end of the day, I don’t know what the right solution is. This “problem” came about due to the meaning of `private` (and then `fileprivate`), and the “simple” approach taken to access control, which caused us to organize code around the file, causing some *very* large files. Some time back, many people proposed solving this problem through additional Access Control modifiers to "fill in the gaps” between `internal` and `fileprivate`. There has been a lot of discussion on that score, and time and again, the consensus seemed to be “This smells like a sub-modules problem, lets solve it there”. Very well, lets. The problem hasn’t gone away (for those of us who have it).

Yes, there seems to be a need for “protected” on type level. With “protected” taken in the usual C++ parlour. (Sub) modules are a way to approach that, but it could be solved with an additional access level as well.

Personally, with the arrival of SPM I no longer have a _need_ for modules. But it is probably still a good idea to have some mechanism to emulate/implement the “protected” access level.

···

On 06 Mar 2017, at 15:10, Karim Nassar <karim@karimnassar.com> wrote:

On Mar 6, 2017, at 3:04 AM, Rien <Rien@Balancingrock.nl> wrote:

For the moment—though we actually propose to call this unit a sub-module—we'll call this new unit a Foo. Let's describe properties of a Foo:

* A Foo should serve to organize one or more File code units by:
- ensuring a File (the code-unit) may belong to one and only one Foo
- making clear (with or without an IDE) to which Foo a given file might belong
* A Foo should have a unique name
* A Foo should be able to nest under another Foo, and the name of that nesting should be clear & unique
* A Foo should be able to be easily created, split, and merged as the codebase evolves
* A Foo should be able to be moved, copied, and SCMed as a unit, without reference to anything outside the Foo

Having now described the critical properties of this new unit called “Foo” have we not also described a filesystem directory?

Many things in life share characteristics. And while we should certainly look to them for lessons, we should not conclude that they are the same.

And that’s also fair. As I said, it was a thought-experiment trying to explain how I got to the directories concept. If it’s wrong, then it’s wrong.

—Karim