Namespacing of packages/modules, especially regarding SwiftNIO

[johannesweiss]

First of all, I apologise for bringing this up only now but I believe this is very important for the long-term health ecosystem and it's not too late yet.

What's the issue?

In a given Swift(PM) application, 3 things about a package need to be unique across the whole application:

1. it's package basename (for example swift-nio-http2)
2. it's package name (let package = Package(name: "swift-nio", ...) in Package.swift)
3. all module names

Arguably, that can lead to tricky issues when combining multiple packages. We realise that SwiftNIO is becoming a dependency of more and more libraries and therefore we want to make your lives easier by defining rules for ourselves on how to name things.

So what is NIO doing to help?

Because all those identifiers (basename, package name, module, and type names) can not be changed without releasing a new public API version we try to make the ones we add predicatable. For SwiftNIO, we have tried to be very careful and we have specified what exactly we will and won't do, and how we namespace new symbols in SwiftNIO's public API declaration.

Basically: Unless we release a new major version, we will only add symbols (in minor version bumps) that have NIO* / nio* / CNIO* / cnio* / c_nio* prefixes. All of our modules are and will always remain NIO*/CNIO* too. And our packages are swift-nio*.

How does this help?

Unless you name your package swift-nio-* or create modules that start with NIO/CNIO and follow the recommendations NIO's public API declaration there should be very little room that your package is clashing with SwiftNIO today or in the future, so it should remain compatible. That is very important for the ecosystem to work long-term when we have an order of magnitude more packages that depend on each other.

Why am I bringing this up?

The reason I bring this up now is that mostly as part of the SSWG, there are a lot of great new packages being created that all use SwiftNIO and we couldn't be more excited about it. Examples are:

• https://github.com/mordil/nio-redis
• https://github.com/vapor/nio-postgres
• https://github.com/kylebrowning/swift-nio-apns
• https://github.com/swift-server/swift-nio-http-client
• https://github.com/openkitten/niodns
• ...

The problem here is that there is a high chance of name collisions in the future because a lot of those packages are either swift-nio-* or have modules/types called NIO*. This will basically break everything we try to help you with by setting out rules for ourselves in the public API guidelines. In other words: If everybody names they modules/types NIO* and their packages swift-nio-*, then breakages are similarly likely to if we just didn't use our namespaces at all.

I believe that we're solving a real issue here and therefore my proposal would be to sort of claim packages prefixed swift-nio as well as module/type names starting with NIO/CNIO as SwiftNIO's namespace. I realise that it's very late asking people to rename their packages/modules to something else and it some cases it won't be possible anymore. By all means, if you have released a public version that people are using, please don't worry to change things until at least you release a new major version. For upcoming SSWG packages however I think it would be great to not use swift-nio*/NIO*/CNIO*.

Does that make sense? Is this too much to ask? And what do you think about this?

CC @tanner0101 / @IanPartridge / @kylebrowning / @Mordil / @artemredkin / @lukasa / @Joannis_Orlandos

[Joannis_Orlandos]

Good point. I haven't considered this impact yet, but I'm completely behind this.

[kylebrowning]

Im cool with this. I think it's important to protect namespace.

[tanner0101 (Tanner)]

I agree protecting the namespace seems important. But for NIOPostgres at least, I really can't think of a better name. Does anyone have ideas? A pattern that we could use consistently would be nice. Maybe a special prefix for NIO-community things?

NIOPostgres is very similar in design to NIOHTTP1, NIOWebSocket, etc. That's why I like the name quite a bit. From my perspective, a user importing something with the NIO* prefix should expect things to be lower level. It's a nice pattern.

Another idea is that maybe we don't need to have such a broad rule. For something like NIOPostgres or NIORedis, where it's very unlikely that Apple will be creating a driver, we can let it slide. Then we only have to worry about things that are more likely to clash, like NIODNS.

[Mordil (Nathan Harris)]

Tanner voices my thoughts exactly. Agree with the goal, but not sure what the alternative is without being cute with the naming, overly broad, or redundant.

[johannesweiss]

@Mordil

Tanner voices my thoughts exactly. Agree with the goal, but not sure what the alternative is without being cute with the naming, overly broad, or redundant.

What about PostgresNIO, RedisNIO for the module names?

[artemredkin]

Same here, what name do you think would be acceptable for NIOHTTPClient? Could we maybe make a registry of accepted libraries and their namespaces (at least for NIO*)?

[tomerd]

/cc @Aciid to see if he ideas from swift-pm point of view

[Mordil (Nathan Harris)]

@artemredkin

Same here, what name do you think would be acceptable for NIOHTTPClient ? Could we maybe make a registry of accepted libraries and their namespaces (at least for NIO* )?

Honestly, for some of these lower level libraries I'm a little willing to claim some very basic module names: HTTP, Redis, etc. while the package might just be as simple as swift-*

I'd very like to see import HTTP similar to other language ecosystems

Higher level libraries will be responsible for naming, such as *Kit or other scheme.

[johannesweiss]

@Helge_Hess1

NIOPostgres

I thought the idea is that once that becomes the official working group implementation, the name would be allowed to enter that namespace. Is it not so?

"Official working group implementation" sounds very formal to me. The SSWG is really just there to help coordinate and mature the ecosystem by running and incubation process. Depending on SSWG packages might also be a slightly easier dependency because you can rest assured that there are people looking after it, at the very least 'archiving' an old obsolete project. Also you can rely on the licensing and just a few more pairs of eyes but that's all, see minimum requirements.

The SSWG also doesn't say 'our implementation for XYZ' is the only one and the best. There could be two packages wanting that implement the same thing. (see ecosystem index).

@Helge_Hess1

For upcoming SSWG packages however I think it would be great to not use swift-nio* / NIO* / CNIO* .

So you anticipate that Apple will "secretly" produce NIO packages directly competing with official "SSWG packages" people are working on?

That's not what I said at all. I'll give you better examples: Let's assume someone creates a module called NIOBasics, or NIOHelpers, or maybe NIOSomethingElse. Or someone adds a type called NIOEventLoopProvider or similar. There's no way we can then guarantee that this package will be compatible with future NIO versions because as we announced, our strategy to prevent clashes is to always prefix everything new (apart from in new major versions) with NIO. That just doesn't help if everybody is creating NIO modules/types.

@Helge_Hess1

Or in other words: I'd prefer the NIO prefix to be community owned

But then we would need to have some registry of stuff using NIO* to be able to safely add modules/types to SwiftNIO, right? Or should SwiftNIO switch to a different prefix in the next major version? What is your proposed fix?

[johannesweiss]

@artemredkin

Same here, what name do you think would be acceptable for NIOHTTPClient ? Could we maybe make a registry of accepted libraries and their namespaces (at least for NIO* )?

NIOHTTPClient is a good example of something I believe we should better rename. NIOHTTPClient is a very near miss to the executable target called NIOHTTP1Client in SwiftNIO.

A registry would of course work but I was hoping we can avoid that complexity by just name-spacing things .

[tanner0101 (Tanner)]

@Mordil

Honestly, for some of these lower level libraries I'm a little willing to claim some very basic module names: HTTP , Redis , etc. while the package might just be as simple as swift-*

For Redis, that may be considered trademark violation. We had that issue with SQLite at least: Trademark Violation · Issue #47 · vapor/sqlite-kit · GitHub

@Helge_Hess1

My suggestion is that the NIO prefix is reserved for SSWG incubated projects, not for Apple.

That seems like a reasonable compromise to me at least.

@Helge_Hess1

Something which I think would be a step backwards and counter the idea if people put vendor names into the released modules. NIOVaporPostgreSQL, NIONozeRedis, NIOKituraXYZ.
I'm in strong favour of using NIOPostgreSQL for a PG NIO driver w/o additional dependencies, or NIORedis, etc.

I agree 100% with this. I definitely want to avoid making it seem like these packages are anything but "pure NIO". In other words, NIOPostgres is, at its core, NIO + Postgres. Adding any other words there, especially the name of a framework, could make it seem like the package is more specialized.

[Mordil (Nathan Harris)]

@tanner0101

For Redis, that may be considered trademark violation. We had that issue with SQLite at least: https://github.com/vapor/sqlite/issues/47

I'm aware of this consideration, and my understanding is the primary concern being the name of the project in GitHub.

How will people be able to easily disambiguate xyz/redis from a fork of the actual Redis project and a Redis client library - which ours are?

Naming it nio-redis (or swift-redis) as a package/repo is fine, and naming the module Redis is fine as it matches ecosystem convention - or something like that.

[Mordil (Nathan Harris)]

After some more thought, I've realized that we already have an established process for naming these packages and modules if we follow what we came out with Logging & Metrics.

We're working in some aspects a bit too hard to get simple names to do double or triple duty in providing information and context.

• NIORedis as a package could become swift-redis and the module just Redis
• NIOPostgres -> swift-postgres / Postgres
• NIOAPNS -> swift-apns / APNS
• NIOHTTPClient -> swift-http / HTTP
  ...

[johannesweiss]

@Helge_Hess1

What is your proposed fix?

My suggestion is that the NIO prefix is reserved for SSWG incubated projects, not for Apple. If a NIOBasics package/module/type got accepted as part of the process, that is it. The WG acts as the registry. The "to help coordinate and mature the ecosystem" sounds very much in line with that.

The SSWG isn't about picking friends or electing one library as 'the only one' for a given task. If the SSWG were a forum to elect the 'best and only' library for a given task, that would be a huge issue. What if someone developed a better library for a task that the SSWG has a library for? Should we ignore that, or kick the existing one or, or force them to merge projects? I think all of those options are bad.
I believe that without knowing the future, it's impossible to figure out what will become the best version of a library and therefore the SSWG explicitly states that there might be more than one project that solves a given project (ordered by popularity).

The guarantees given by an SSWG-approved project are at least the ones listed under minimum requirements. Therefore I don't think we should give SSWG-approved packages any magic special powers. I think it's rather the opposite, for SSWG-approved packages we can require a certain set of rules (the minimum requirements), for example regarding licensing. Nobody can stop you from creating a GPL3 licensed driver for a database but the SSWG requires 'Apache 2 by default' (into which I personally read that it should be Apache 2 unless there are good reasons and even then it should be a permissive license comparable and compatible to Apache 2).

@Helge_Hess1

If someone else (like myself) also does a NIOxyz outside of the WG, that would be entirely my problem (though I would probably just blame Swift/SPM )

Of course you can do that and maybe later at some point you might even want to pitch NIOxyz to the SSWG? The only issue with that would be if everybody who starts writing a 'xyz' client or server names their package and module NIOxyz. Then we have what nobody wants: A set of mutually exclusive 'xyz' implementations.

Please don't get me wrong, this isn't about giving any project (including NIO) special powers, this is about making sure that as many packages as possible can be mixed with others without clashes. For example, let's assume you create a package nio-foo and somebody else creates swift-nio-foo, and both would have a NIOFoo module. At some point, one of you two pitches say nio-foo to be added to the SSWG, it gets approved, and then some of the very popular big web frameworks starts using nio-foo somewhere in their dependency tree. Now, the community has a problem because it's now impossible for anybody to use that popular web framework together with swift-nio-foo which is a real bummer.
Worse, the problem won't be hit by the authors of those libraries, it will be hit by some developer who (possibly long ago) chose swift-nio-foo as their favourite Foo library and a certain poplular web framework as their favourite web framework and at some point that just stops working (because the web framework pulls in the other Foo library).
We'll run into similar issues with packages that provide higher-level APIs for stuff that NIO covers or will cover (HTTP/1&2, SSL/TLS, WebSocket, QUIC, DNS, UDP, TCP, event loops, ...) or very general names. For example we recently added the NIOTestHelpers module to make it easier for developers to verify their NIO code in their own unit tests. If anybody else created such a module in say their database driver, we're in trouble I think.

@Helge_Hess1

There could be two packages wanting that implement the same thing

While I'm not entirely sure why this is desirable at all, I'd say it's first come first serve in this case.
In any case there is no good solution for this in any case due to the lack of proper namespacing in Swift (e.g. via DNS names). What would happen? One imp would call itself RedisNIO, well, and the other too!

I don't think this is desirable either but it might happen. Joining efforts to make one great library is certainly better than creating lots of half-baked alternative versions.
And you're right naming a Foo package just FooNIO (instead of NIOFoo) isn't magically solving all problems because two packages might do the same.

I think I opened this discussion way too NIO-centric (because that was the concrete problem I was worried about and well because I work on NIO ). Maybe a better of phrasing the question is:

Should we come up with a certain set of best practises around package/module/type naming and name-spacing? I believe that yes and I reckon the only way to do this is that when a new project is created, github/gitlab/... are searched for potential clashes and we make an active effort not to reuse anybody's package (base)name or their module prefixes.

@Mordil

After some more thought, I've realized that we already have an established process for naming these packages and modules if we follow what we came out with Logging & Metrics.

We're working in some aspects a bit too hard to get simple names to do double or triple duty in providing information and context.

• NIORedis as a package could become swift-redis and the module just Redis
• NIOPostgres -> swift-postgres / Postgres
• NIOAPNS -> swift-apns / APNS
• NIOHTTPClient -> swift-http / HTTP
  ...

I agree with that but we need to be careful to not create other clashes. For example I bet there are already lots of packages called HTTP out there. And also: Should this be for an HTTP client or server? Same for Postgres, APNS, and Redis just not quite as bad as HTTP.

As above: I think we should aim to make package/module names unique globally (and not only in the SSWG) if possible and also foresee extensions (after a HTTP client, there might be a HTTP server, ...). This all is not bullet proof because someone could also later start a new project with the same name as an existing one. But as a set of best practises, what about trying to

• come up with a package (base)name that's unique globally if possible
• not hogging anyone else's module prefix if possible

These are not meant as absolutes but as something we could aim for. Of course, I know this is not easy, I just wanted to talk about the issue of clashes which will become more and more real the more the ecosystem matures as we'll likely share much more code than we used to (which is great!).

[Mordil (Nathan Harris)]

@johannesweiss

The SSWG isn't about picking friends or electing one library as 'the only one' for a given task. If the SSWG were a forum to elect the 'best and only' library for a given task, that would be a huge issue.

@johannesweiss

Therefore I don't think we should give SSWG-approved packages any magic special powers.

This is definitely in the charter, and I think was a community misinterpretation in some regards. I also think it's inevitable due to human nature to see a committee that has recommendations and processes behind them to be seen as an Authority Body and the recommendations being "official" in nature, even not by name.

@johannesweiss

this is about making sure that as many packages as possible can be mixed with others without clashes.

I'm not sure of the value of this? If I see two libraries that implement a Redis client - I would assume they're mutually exclusive, even if they're not to the build tools. I probably wouldn't want to use both in my library / application for making clients, and at some point I would have to write adapter code to even use parts that I might want from RedisLib-1 with RedisLib-2.

@johannesweiss

Worse, the problem won't be hit by the authors of those libraries, it will be hit by some developer who (possibly long ago) chose swift-nio-foo as their favourite Foo library and a certain poplular web framework as their favourite web framework and at some point that just stops working (because the web framework pulls in the other Foo library).

I thought the purpose of a framework was to rely on them to pull on the disparate libraries so that you don't have this distinction? Otherwise, they're in no different of a situation than web developers when say, choosing between React, Vue, or Angular

@johannesweiss

I believe that yes and I reckon the only way to do this is that when a new project is created, github/gitlab/... are searched for potential clashes and we make an active effort not to reuse anybody's package (base)name or their module prefixes.

I think having a standard for naming among current or upcoming SSWG libraries is a good idea - but to have it completely unique is extreme overkill. We'll end up with cute or overly long/verbose names rather than very helpful and descriptive names.

I think what we're really getting at in this discussion is that there are problems with Swift tooling in regards to package composability (which, personally, that aspect of the Swift project has been neglected) and it's an uphill battle.

[Mordil (Nathan Harris)]

@johannesweiss

Should this be for an HTTP client or server

As a minor note - my hope is that I would be able to have a Client, and server(s) available from the module eventually.

HTTPClient, HTTP1Server, etc all vended by the HTTP module

[tanner0101 (Tanner)]

@Mordil

I think having a standard for naming among current or upcoming SSWG libraries is a good idea - but to have it completely unique is extreme overkill. We'll end up with cute or overly long/verbose names rather than very helpful and descriptive names.

I think what we're really getting at in this discussion is that there are problems with Swift tooling in regards to package composability (which, personally, that aspect of the Swift project has been neglected) and it's an uphill battle.

@Helge_Hess1

Say if there is a "wrong" NIORedis already, I'd need to rename mine to NIORedisBest.
That sounds like a reasonable requirement for a pitch.

@Helge_Hess1

I would tend to say no, it is no issue right now and the real thing needs to be solved/fixed in SPM/Swift by doing proper namespacing.

I agree with these thoughts. I do think it's a valid concern that we could have conflicting module names in the future, but I don't think there's much we can do about it without being pedantic. The real solution is for SPM / Swift to get proper namespacing.

If a package being pitched has an exceptionally high chance of clashing in the future (such as NIODNS perhaps), we should take that as a special case during the proposal process. But for things like NIOPostgres, NIORedis, etc, where there is very little chance of Apple wanting to release their own version, I don't think we need to worry about them. Even if Apple were to release github.com/apple/swift-nio-postgres, I think that should probably replace GitHub - vapor/postgres-nio: 🐘 Non-blocking, event-driven Swift client for PostgreSQL. anyway. And, like Helge said, if someone else wants to make their own NIO + Postgres package, it's up to them to pick a unique name since NIOPostgres will have already been taken.

[johannesweiss]

@Mordil

this is about making sure that as many packages as possible can be mixed with others without clashes.

I'm not sure of the value of this?

@Helge_Hess1 covered this one really well. I think it is very important to support this use-case.

@Mordil

I believe that yes and I reckon the only way to do this is that when a new project is created, github/gitlab/... are searched for potential clashes and we make an active effort not to reuse anybody's package (base)name or their module prefixes.

I think having a standard for naming among current or upcoming SSWG libraries is a good idea - but to have it completely unique is extreme overkill.

Completely unique is not achievable but it's something we can aim for. And I'm arguing we should.

@Mordil

Should this be for an HTTP client or server

As a minor note - my hope is that I would be able to have a Client, and server(s) available from the module eventually.

HTTPClient , HTTP1Server , etc all vended by the HTTP module

I don't think this works well because then the major versions of two completely separate things are tied together inseparably. So an already mature HTTP client library might need to major version +1 just because the HTTP server needs a major rev. Especially when we're starting with one component and doing the other later, this would lead to real issues.

@Helge_Hess1

If I would pitch that to the SSWG (though my projects don't qualify in the first place),

Your packages absolutely qualify just like everybody elses. Any particular requirement that you feel you don't qualify for?

@Helge_Hess1

I would need to make sure that my names do not clash with other package names. Say if there is a "wrong" NIORedis already, I'd need to rename mine to NIORedisBest.
That sounds like a reasonable requirement for a pitch.

100% agreed. In other words: Everybody who's pitching something to the SSWG should make sure that there are no clashes. This IMHO should be added to the minimum requirements, I'm planning on pitching this soon enough.

@Helge_Hess1

(though I thought that the whole point is stopping all that duplicated work by providing base modules that work for everyone [e.g. I appreciate that Tanner is trying to do this with NIOPG])

What the SSWG wants to help create is an awesome ecosystem where we can all share a lot of code and reduce costly duplication. Ideally that way we get a broader set of good libraries rather than just 10 half-done drivers for the same DB. However, there might be a point where an existing library is just not good enough and can't be repaired without completely changing everything and in those cases we need to be able to let people use both. In some cases there might also be competition between two already existing libraries and the SSWG is not electing the winner. If both qualify, then both should be approved. Again, working together might lead to better results but we can't or won't force two or more parties to collaborate.

@Helge_Hess1

I would tend to say no, it is no issue right now and the real thing needs to be solved/fixed in SPM/Swift by doing proper namespacing. My understanding is that the infrastructure for doing it "right" is available, it "just" needs to be done ;-)

I haven't even seen an evolution pitch yet to introduce something like this. I believe we can't wait for that to happen and in theory it's just strings, we could all name our packages com_apple_SwiftNIO and call it a day . Well, we'd need better qualified imports and such...

@Helge_Hess1

In the meantime I think we are good. Did we ever had such clashes already?

Yes, two out of two Redis libraries share the same module name:

• https://github.com/SwiftNIOExtras/swift-nio-redis/tree/develop/Sources/NIORedis
• https://github.com/Mordil/nio-redis/tree/master/Sources/NIORedis

and are therefore incompatible. Vapor 3 can't use swift-log at the moment because both have a Logging module and there are also two swift-nio-http-client projects (in this case, the second one is now archived so we're probably good). Internally we had some more and that's just the cases I know about and we're really just getting started!

@Helge_Hess1

If the dep trees get so complicated that this is becoming an issue, it is more of a sign that we must have proper language support for vendored packages (e.g. de.zeezide.noze.redis.NIORedis and com.blub.mongol.NIORedis).

@tanner0101

I agree with these thoughts. I do think it's a valid concern that we could have conflicting module names in the future, but I don't think there's much we can do about it without being pedantic. The real solution is for SPM / Swift to get proper namespacing.

I don't think we should wait for language changes that aren't even pitched and might never arrive. What's wrong with doing some due diligence making sure there are no clashes by searching github, making predicatable names (such as basename == package name and a simple scheme for module names), and also giving certain namespace prefixed to certain projects like people have been doing in (Obj)C for a long time?

@tanner0101

But for things like NIOPostgres , NIORedis , etc, where there is very little chance of Apple wanting to release their own version, I don't think we need to worry about them.

This is totally not about Apple and top-level features or projects Apple or NIO might want to add. My personal opinion is just that we should have very simple guidelines on what's recommended and what isn't. I just think it's a bit too complicated saying NIOPostgres & NIORedis are okay but NIOHTTPClient, NIODNS, NIOHelpers, NIOUtils, NIO... are not. But if that's what everybody thinks is best, cool, let's go with that.

I just think it would be simpler if we just avoided certain 'namespaces' and that is totally not only about NIO. Like I don't think anybody should create a FluentHelpers module because you might want to add that to Fluent later and that might at some point clash with something else. I used the NIO example because NIO already has so many modules and more and more are added to other packages. Please don't get me wrong, I love that so much stuff is created with NIO, it's really awesome. Also it won't be NIO code itself that fails compilation, it will be some poor developer who happens to combine two libraries that nobody has combined before. Basically the person furthest away from the ones who created the problem has to deal with weird compiler errors that the developer can't easily resolve. I think we as library authors should try to prevent these problems as much as possible . Of course within reason but I believe we should try. That's why in NIO we tell you today the prefix of stuff we might be adding tomorrow. So if you never want to clash with anything NIO, avoid NIO* and that's it. An easy rule that I believe provides value but I'm also cool if the community thinks we're discussing a non-issue.

[johannesweiss]

@Helge_Hess1

Yes, I think I remember that 1-developer packages do not qualify (and I don't actually question that rule, it makes sense).

You might accept contributions though. So if there's a package that has only yourself as a contributor today, you can totally pitch that which might also attract new contributors. Also we have a 'sandbox' level which is a great place for a package to get started.

@Helge_Hess1

So if you never want to clash with anything NIO, avoid NIO* and that's it.

Except that doesn't really solve anything but just give Apple freedom to introduce arbitrary NIO packages (a mini-mini registry). It will just result in RedisNIO vs RedisNIO and solve 0 issues for the "poor developer".

Apple's got a great marketing team, I'm sure they can find creative ways . Joking aside, trust me, this is really not the reason for me (personally) proposing to create namespaces, nobody can stop you from creating clashing packages anyway.

Regarding RedisNIO: I personally think that would be better because it IMHO lowers the possibility of clashes. And just like you proposed above, the second RedisNIO package should just pick a better name, for example BestRedisNIO or just BestRedisClient.

Again, we can't enforce any of this but we, the community, might want to create a set of guidelines. For the SSWG (with a proposal to change the minimum requirements) could then even formalise the rule: "SSWG packages shouldn't pick existing repo/module names.".

One other possibility would be to say "within reason, a module name spawns a namespace". In other words after somebody creates RedisNIO, another package shouldn't just create RedisNIOClient without at least checking with the original RedisNIO folks. Again, all within reason, there will also always be accidental clashes etc.

But it seems like most people don't seem to agree with me that namespaces are useful and that's okay, let's not do this then. Maybe we should focus on at least unique module/package names.