"Submodule" is an ambiguous term

A request from me: if you use the word "submodule" to refer to any particular feature or design, please explain what you're using it for. People have a lot of different features they want in Swift that could be called "submodules", and there's not one agreed-upon design that people will understand when you say the word. I'll try to list the most common requests / proposals below. (Note that many proposed designs have combined more than one of these features. I've tried to decompose them for the purpose of this discussion, but they're not inherently independent.)

Namespaces

namespace Foo {
  func foo() {}
}
namespace Foo {
  func test() {
    foo()
  }
}

// Out here I have to qualify my reference, like a static function.
Foo.foo()

Namespaces come up fairly often as a request even without mentioning the word submodules. They're a way to group things and to keep from cluttering the top-level identifier space. Today the recommended idiom is to use a caseless enum:

enum Foo {}
extension Foo {
  static func foo() {}
}

This has nearly the same effect, but isn't really distinguishable from a normal enum where the coder just hasn't added any cases yet, or a local version of Never.

Some additional features namespaces could have:

  • private would probably behave like it does in types, making a declaration only accessible within that namespace in the same file.

  • C++ has a feature called using that allows you to use names from a particular namespace unqualified within a particular context. Other languages have similar features.

Access Control

A few people have started up discussion on this again, and so I don't want to get into it on this thread, but if you grouped a handful of files together and called that a submodule it might be useful to have an access control level between internal and fileprivate that means "these files, but not the entire module". This gets trickier if you allow nesting or overlapping of these groups, or if you try to do it with declarations rather than whole files, but there could be something here.

(Compare with Rust's access control model, where something is either "exported to the next level up" or "only exposed to me". There are additional features too but I get the impression those are less common.)

Dependencies

Since Swift's incremental build logic is still *cough* conservative, some people have proposed grouping files together for dependency analysis purposes. I'm not quite sure how this works; does it mean they don't rely on the rest of the module? Or does it mean that the rest of the module doesn't rely on them? Or is there some kind of "local import" to explicitly declare such dependencies? But it could be useful.

Explicit imports

A feature that Clang's modules have is the ability to separate off some APIs and only expose them if you spell out the full import path. The most common example of this is UIKit.UIGestureRecognizerSubclass. If you use import UIKit, you get most of UIKit's submodules: UIKit.UIView, UIKit.UIGeometry, etc. (Most submodule names are inferred from the headers included by the umbrella header.) But you don't get the parts of UIGestureRecognizer needed to make a subclass, because most of the time they'd just be in your way. You only get them if you say import UIKit.UIGestureRecognizerSubclass.

It's not exactly clear how you'd define these in Swift.

(Note that this wouldn't actually provide much safety because of other problems: today, extension methods are made visible everywhere if they're imported anywhere.)

Multiple modules, one target

The Swift compilation model says that all code in one module is compiled together. Sometimes people want "sub-target modules" instead: something that acts like a module at the language level, but which is still built into a single binary output. (Static libraries do most of this, especially if we formalize @_exported, but not the optimization part.)

A related thing people want is to only expose certain declarations to other modules in the project / package. This is kind of "access control", kind of "explicit imports", and kind of "multiple modules, one target", so I'm sticking it here. (It's also almost never called "submodules", so maybe it doesn't belong in this post at all, but whatever.)

etc.

I think that covers the top features that people have called "submodules". If you're going to propose any of them, please be explicit about what you are and aren't including!

Some additional thoughts for any such proposal:

  • Are you grouping files or declarations? If declarations, do all the declarations have to be in the same file or not?
  • Can you nest submodules? Can one file or declaration be part of multiple submodules?
  • What are the ABI and source compatibility implications?
  • How does it interact with access control?
  • What does it mean to have an extension of someone else's type in your submodule?

EDIT: See also The One Stop Shop for Previous Submodule Pitches.

16 Likes

Thanks for writing this up, Jordan.

One thing I want to clarify about "multiple modules, one target" is that, unlike most of the rest of these ideas, it's really a tool feature, not a language feature. That is, nothing in the source code of a module should care whether it's being built as a separate dynamic library or bundled with several other modules into a compound dynamic library or just directly linked into an executable. The user just specifies at some higher level that they want to build it that way and the code gets optimized / linked better.

That doesn't mean it's not appropriate to talk about in Evolution; it just means that it should be an explicit goal of that effort to not require changes to the module source.

Does that hold when there are supposed to be groupings/submodules that can be imported? For example, If I only wanted to import iOS playback, wouldn't I have to indicate in my code, somewhere, what types are in what modules?

  • MyMusicLib
    • Model
    • Playback
      • iOS
      • macOS
    • Analysis

Amazing write up Jordan, it deserves its own Swift.org blog post hehe

(Assuming you're responding to me)

I'm not quite sure what you're getting at, but I think it might be confusion from the unfortunate overloading of the word "target". Jordan and I are using "target" to mean a final build product: an executable or dynamic library. I think you're talking about the "target" in the sense of the architecture and OS where those build products are expected to run.

Information like "what types are in what modules" arises from the module structure, not from the build-target structure. My point is that there's a directed graph of modules (source-level bundles of code) and their dependencies, and while that graph has to be reflected in the directed graph of images (dynamic libraries or executables, i.e. runtime-level bundles of code) and their dependencies, it doesn't need to be identical to that graph because you can reasonably bundle separate modules together into a single image, as long as all their dependencies are either in that same image or in a dynamic library it depends on.

So yes, you make a source-level decision to import some module or another, but you don't care at the source level how that module is made available at runtime; you just know that it's a separate module that you can import.

1 Like

Thank you. I meant how would I make it clear that those groups exist to be imported independently/instead of MyMusicLib? I am thinking of some syntax to say module Model exists within the main module MyMusicLib.

That would be a source-level feature of module nesting, which is not part of the bullet point I was talking about.

1 Like

@jrose thank you for this clarification, I have one question though. If you say in the section about access control that a submodule modifier should be in-between internal and fileprivate do you imagine such an access modifier to be applied on type declarations as well or rather on type members? Former is indeed tricky and probably very hard to express. This was a general question to understand your direction, so don't get to much into detail. It is not my intention to mutate this thread into another access control thread.


One more general question to this thread: Is it reasonable that Swift could potentially adopt multiple "submodule" approaches? I think this direction is yet not clear nor do I remember a decision made in that regard.

Anyone coming up with such a proposal would have to decide that in their proposal. And yes, I think it's possible for the language to support more than one of these features, but only one of them would be called "submodules".

4 Likes

@Honza_Dvorsky and I were talking about something else in the "multiple modules, one target" space, which is when you have a Swift package with a bunch of different modules in order to enforce separation of concerns, intra-package layering, whatever, but where clients of the packages should always interact with the top-level module. This is similar to what I originally wrote (particularly about formalizing @_exported), but it's a little different because in this case you'd rather not expose the module name of these "sub-package implementation detail modules" outside of the package itself. (Ideally you wouldn't include it in the mangling either, so that you can move entities around without changing the ABI, but I'm not sure how to do that while still avoiding collisions.)

2 Likes

@jrose it’s great to know that you’re continuing to think about this topic. :tada:

This is exactly the use case I think “submodules” should aim at addressing. It is a really important use case (especially in larger code bases) and is completely unserved by Swift today. Given the meaning of “module” in Swift today, “submodule” seems like the appropriate name for a feature in this space: something much like a module but without the overhead of an independent build target.

Related use cases include making some of these intra-target submodules visible to clients, allowing selective import, etc, but these features are ancillary to the fundamental concern of allowing programmers to express intra-target structure in a layered fashion.

Other use cases that are sometimes discussed when “submodules” come up are really more like namespaces. It isn’t yet clear to me that features in this direction provide sufficient benefit beyond caseless enums to justify their presence. That said, if sufficient motivation is presented I think a name other than “module” or “submodule” for a feature in this area would make the most sense.

So to summarize, +1 to thinking in the direction of your post. I would love to see this area explored in Swift 6 if possible.

3 Likes

After reading your post I wonder if namespaces would simplify the impact on the type system, since the compiler probably would no longer need to do any enum related checks anymore. Would be interesting to know, even just in theory. So maybe someone with more insights could answer that.

It'd be slightly faster to compile and slightly smaller in terms of code size, yeah. I wouldn't expect a significant difference, but the minor benefits are there.

2 Likes