If I would pitch that to the SSWG (though my projects don't qualify in the first place), I would need to make sure that my names do not clash with other package names. Say if there is a "wrong" NIORedis already, I'd need to rename mine to NIORedisBest.
That sounds like a reasonable requirement for a pitch.
As you like to say: That's not what I said. The SSWG would only make sure that there is no overlap in the module/package names. We can still have 10 Redis clients. (though I thought that the whole point is stopping all that duplicated work by providing base modules that work for everyone [e.g. I appreciate that Tanner is trying to do this with NIOPG])
I would tend to say no, it is no issue right now and the real thing needs to be solved/fixed in SPM/Swift by doing proper namespacing. My understanding is that the infrastructure for doing it "right" is available, it "just" needs to be done ;-)
In the meantime I think we are good. Did we ever had such clashes already? If the dep trees get so complicated that this is becoming an issue, it is more of a sign that we must have proper language support for vendored packages (e.g. de.zeezide.noze.redis.NIORedis and com.blub.mongol.NIORedis).
But even that is not really sufficient, we can't even carry multiple versions of the same package, which is actually an even bigger real world issue.
I don't know whether any other current language but ObjC is also affected by that. Java, JS, Python even C all have working solutions for that. Maybe Ruby? Don't know ;-)
I agree with these thoughts. I do think it's a valid concern that we could have conflicting module names in the future, but I don't think there's much we can do about it without being pedantic. The real solution is for SPM / Swift to get proper namespacing.
If a package being pitched has an exceptionally high chance of clashing in the future (such as NIODNS perhaps), we should take that as a special case during the proposal process. But for things like NIOPostgres, NIORedis, etc, where there is very little chance of Apple wanting to release their own version, I don't think we need to worry about them. Even if Apple were to release github.com/apple/swift-nio-postgres, I think that should probably replace GitHub - vapor/postgres-nio: š Non-blocking, event-driven Swift client for PostgreSQL. anyway. And, like Helge said, if someone else wants to make their own NIO + Postgres package, it's up to them to pick a unique name since NIOPostgres will have already been taken.
I think this assumption is wrong, things like that do happen especially if the packages are lower level and not registered. It is common in many environments to carry multiple versions and implementations of the same lib (in Java, in JS, on Windows even the C runtime/stdlib!!)
For example a user might want to mix a Kitura user management middleware with an El Fumar session tracking one. Both might use Redis and use distinct Redis client libs (and that should be fine).
Or lets say a Kitura server does some HTTP client calls using some HTTP client library. But it also wants to do APNS and uses some other package for that. That the two are using the same HTTP lib is not a given (right now even unlikely).
Those issues need to be fixed in SPM, not by declaring the NIO prefix private. The latter really doesn't solve anything at all (RedisNIO vs RedisNIO).
So I stick to my recommendation to make the SSWG kinda a mini registry until a proper namespacing solution is provided (Swift 8?).
@Helge_Hess1 covered this one really well. I think it is very important to support this use-case.
Completely unique is not achievable but it's something we can aim for. And I'm arguing we should.
I don't think this works well because then the major versions of two completely separate things are tied together inseparably. So an already mature HTTP client library might need to major version +1 just because the HTTP server needs a major rev. Especially when we're starting with one component and doing the other later, this would lead to real issues.
Your packages absolutely qualify just like everybody elses. Any particular requirement that you feel you don't qualify for?
100% agreed. In other words: Everybody who's pitching something to the SSWG should make sure that there are no clashes. This IMHO should be added to the minimum requirements, I'm planning on pitching this soon enough.
What the SSWG wants to help create is an awesome ecosystem where we can all share a lot of code and reduce costly duplication. Ideally that way we get a broader set of good libraries rather than just 10 half-done drivers for the same DB. However, there might be a point where an existing library is just not good enough and can't be repaired without completely changing everything and in those cases we need to be able to let people use both. In some cases there might also be competition between two already existing libraries and the SSWG is not electing the winner. If both qualify, then both should be approved. Again, working together might lead to better results but we can't or won't force two or more parties to collaborate.
I haven't even seen an evolution pitch yet to introduce something like this. I believe we can't wait for that to happen and in theory it's just strings, we could all name our packages com_apple_SwiftNIO and call it a day . Well, we'd need better qualified imports and such...
Yes, two out of two Redis libraries share the same module name:
and are therefore incompatible. Vapor 3 can't use swift-log at the moment because both have a Logging module and there are alsotwoswift-nio-http-client projects (in this case, the second one is now archived so we're probably good). Internally we had some more and that's just the cases I know about and we're really just getting started!
I don't think we should wait for language changes that aren't even pitched and might never arrive. What's wrong with doing some due diligence making sure there are no clashes by searching github, making predicatable names (such as basename == package name and a simple scheme for module names), and also giving certain namespace prefixed to certain projects like people have been doing in (Obj)C for a long time?
This is totally not about Apple and top-level features or projects Apple or NIO might want to add. My personal opinion is just that we should have very simple guidelines on what's recommended and what isn't. I just think it's a bit too complicated saying NIOPostgres & NIORedis are okay but NIOHTTPClient, NIODNS, NIOHelpers, NIOUtils, NIO... are not. But if that's what everybody thinks is best, cool, let's go with that.
I just think it would be simpler if we just avoided certain 'namespaces' and that is totally not only about NIO. Like I don't think anybody should create a FluentHelpers module because you might want to add that to Fluent later and that might at some point clash with something else. I used the NIO example because NIO already has so many modules and more and more are added to other packages. Please don't get me wrong, I love that so much stuff is created with NIO, it's really awesome. Also it won't be NIO code itself that fails compilation, it will be some poor developer who happens to combine two libraries that nobody has combined before. Basically the person furthest away from the ones who created the problem has to deal with weird compiler errors that the developer can't easily resolve. I think we as library authors should try to prevent these problems as much as possible . Of course within reason but I believe we should try. That's why in NIO we tell you today the prefix of stuff we might be adding tomorrow. So if you never want to clash with anything NIO, avoid NIO* and that's it. An easy rule that I believe provides value but I'm also cool if the community thinks we're discussing a non-issue.
IMO that's first of all premature optimisation and the wrong fix as well. The only proper fix can be done in Swift/SPM itself.
Except that doesn't really solve anything but just give Apple freedom to introduce arbitrary NIO packages (a mini-mini registry). It will just result in RedisNIO vs RedisNIO and solve 0 issues for the "poor developer".
You might accept contributions though. So if there's a package that has only yourself as a contributor today, you can totally pitch that which might also attract new contributors. Also we have a 'sandbox' level which is a great place for a package to get started.
Apple's got a great marketing team, I'm sure they can find creative ways . Joking aside, trust me, this is really not the reason for me (personally) proposing to create namespaces, nobody can stop you from creating clashing packages anyway.
Regarding RedisNIO: I personally think that would be better because it IMHO lowers the possibility of clashes. And just like you proposed above, the second RedisNIO package should just pick a better name, for example BestRedisNIO or just BestRedisClient.
Again, we can't enforce any of this but we, the community, might want to create a set of guidelines. For the SSWG (with a proposal to change the minimum requirements) could then even formalise the rule: "SSWG packages shouldn't pick existing repo/module names.".
One other possibility would be to say "within reason, a module name spawns a namespace". In other words after somebody creates RedisNIO, another package shouldn't just create RedisNIOClient without at least checking with the original RedisNIO folks. Again, all within reason, there will also always be accidental clashes etc.
But it seems like most people don't seem to agree with me that namespaces are useful and that's okay, let's not do this then. Maybe we should focus on at least unique module/package names.
FWIW On this point, Im a 1 developer for NIOAPNS, but if you get sponsorship for sandbox, you can find the other developer in time to get to graduated. And you won't graduate until your module is proven in the wild by people who are not you.
Well, no. Your point is absolutely valid and I think everyone agrees. Indeed I think most people so far agreed that the SSWG should indeed act as a mini registry. Which I think is a good thing and outcome of the discussion?
P.S.: Also remember that I only did SwiftNIOExtras because there wasn't such an effort (pure NIO modules) at the time! With the current SSWG it may very well be superfluous.
The SSWG doesn't actually have just one place for repos. swift-nio-http-client is mostly an exception because multiple entities are pitching it together so we thought let's use swift-server. But you can totally pitch to the SSWG with a repo in SwiftNIOExtras (your sponsor would need write access to the repo for the 'just in case' case )
One thing I wanted to add here is that it's totally feasible that SwiftPM could either get a more namespaced naming scheme, I don't think this is the main issue. I think the main issue we're facing today is module names and that's mostly a Swift (and not a SwiftPM) issue.
Unfortunately, solving the Swift issue (other than us constructing module names that just don't clash) will be a very very tough sell because it's super ABI relevant and at least on Darwin (mac/i/tv/watchOS) Swift has a stable ABI now.
In case you're interested why this very much ABI relevant: In any language, every symbol (that's functions, constructors, computed properties, meta types, ...) needs a unique symbol name. That's just how linkers work: A symbol is just a string and at least the public ones need to be unique. You might have heard of name-mangling and name-mangling is basically mapping Swift symbols into strings. Those strings are created in a way that they only clash if they're actually referring to the same thing (ie. the same function with the same arguments/return type, in the same type, in the same module, ...). However, 'same module' means 'same module name'.
If you were to create a module called MyModule which contains MyClass which contains func myFunction() -> Void, then that name-mangled string for this function would be
_$s9MyPackage0A5ClassC10myFunctionyyF
If you demangle that with
echo '_$s9MyPackage0A5ClassC10myFunctionyyF' | swift demangle
you'll get back
MyPackage.MyClass.myFunction() -> ()
And symbol names are very much part of the ABI.
Of course, for SwiftPM packages the ABI doesn't matter at all (because we always compile from source) and we could have a special name-mangling scheme for SwiftPM packages that takes more information into account (for example some package identifier) than the information it uses today. But this would be a massive undertaking I believe and very much had to go through evolution.
And btw, this problem isn't unique to Swift at all. In C++ if you were to create a namespace MyNamespace with MyClass that contains myFunction, you'll get for example
That's how the Swift/C linker usually works. E.g. you can load C modules w/ conflicting symbols just fine (e.g. this is how Apache deals with the same issue). On Darwin we even have 2-level namespaces builtin to deal w/ similar issues more easily. JS and Python are other examples which deal with that in a completely different way.
Just as a sidenote, I agree that it'll be hard to change that now, and it is probably neither necessary nor desirable ;-)
The way I imagine that to work is what Java does. It works there forever. People import properly qualified packages w/ IDE assistance, e.g.:
import org.getobjects.appserver.core.WORequest;
I don't see why this wouldn't work in Swift. Clang modules also support such hierarchies already (the C modules below Cocoa are structured in trees already).
As a nice add-on SPM (and Xcode) could shorten names of imported packages. Let's say I import SwiftObjects, which has the module name (bundle-id ...) de.alwaysrightinstitute.SwiftObjects. Within the package I might just say "import SwiftObjects" and SPM would add a proper search path which expands the thing to the full module name.
I'm a fan of consistent naming, I think it helps with discoverability and "guessing" the right thing.
What if we're mixing up namespaces a bit? For projects under NIO, those packages use the NIO/swift-nio- prefix. For projects brought under the SSWG, they use SS/swift-server (or something similar)?
A project under SS to me suggests it not only uses NIO but other foundational libraries like metrics/logging.
You can get it to work with shared libraries in C if you're really careful. As you say, certain systems use that to load modules but usually those modules have a fairly tight and specified 'plugin API'.
But it doesn't solve our issue so I think we're digressing here.
I know you're aware of this, but for other people's benefit:
$ echo -e '#include <stdio.h>\nvoid foo() { printf("foo\\n"); }' > a.c && echo -e '#include <stdio.h>\nvoid foo() { printf("bar\\n"); }\n int main() { foo(); }' > b.c && clang -c a.c && clang -c b.c && clang -o out a.o b.o
duplicate symbol '_foo' in:
a.o
b.o
ld: 1 duplicate symbol for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Yes, but this also does not solve the issue unless we made everything shared libraries and made sure that there's never one module that calls into both conflicting declarations. IMHO linker hacks are not a solution to namespacing, they can be used to work around isolated issues such as two different frameworks both having a private/unexported dependency with conflicting symbol names.
I've not followed the technical discussion closely, but should the SSWG host a database or even github repository where library authors could register a prefix? The SSWG could adjudicate (presumably rare) disputes or (presumably somewhat more common) cases of prefix squatting by defunct projects. A lint rule to enforce the prefix within a project would be a valuable shared resource for library authors.
Thanks for pitching in! Yes, we definitely could and it would solve those issues but I'd have hoped that we don't have to.
My hope would be that we good come up with a 'good enough' scheme to prevent clashes without having to create an explicit registry ourselves. Of course that doesn't scale magically forever but maybe at some point, SwiftPM will support a package registry, maybe with Github's new Package Registry feature?
Yes, I donāt think . would work for us (I havenāt checked but I believe thereās some sort of half working half not-working clang sub modules thing.) so we would need to go for _ or so. So itād be
import com_apple_NIO
Or
import com_apple_SwiftNIO
I think this will work just fine for module clashes. But because we donāt have fully working qualified imports, we would still have issues with the names of the actual types/methods.
Letās say NIO adds a type (as of our rules, we prefix all new types in minor versions with NIO) NIOConnectProxyUpgrader and you have hypothetical code which is
import (com_apple_)NIO
import (com_example_)NIOHTTPProxy
let proxyHandler = NIOConnectProxyUpgrader
Assuming the hypothetical com_example_NIOHTTPProxy module had NIOConnectProxyUpgrader before NIO introduced it in a minor then the code works until NIO gets updated and then breaks. So again a simple clash that breaks software.
I agree this fully solves the module clashes issue but I didnāt propose this scheme in a serious way because I was assuming that the community will just hate it (because naming modules that way is just very non-standard in Swift). Personally Iām fine with that
I don't think I've been strongly pushed in either direction of either adopting a standard or keeping the status quo.
In specific regards to namespace / prefixes for types:
What we've talked about thus far are hypothetical futures, and I believe that it could remain that way, at least for SSWG pitch/proposal packages, is that we're stringent in our feedback and discussion threads to guard against these types of names without them being namespaced such as Redis.foo.
This'll probably be highly important for say, the NIODNS especially as the SwiftNIO team has expressed interest in ingesting it into the base layer of our stacks.
But for something like Postgres and Redis, we've done a well enough job of our types being <Database Name>* or NIO<Database Name>*.
Ultimately, the only inclination I'm having is "we keep a close eye on it, and revisit as needed" - but anything further might be premature optimization.
. Well, we'd need better qualified imports and such...