I've recently been working on bringing various ecosystem components to Windows. Aside from CI still being a lot of work, the majority of libraries are all compiling and running on Windows now with PRs submitted containing these patches.
The only hurdle I came across on Windows is that zlib is not available on Windows by default. It's possible to download it separately through something like vcpkg, but users still need to manually tell the linker where to find these libraries. On top of that, they'll need to store zlib1.dll next to their .exe prior to running the compiled executable. Given the importance of zlib for protocol level compression among others, I think it's quite a tough problem not to fix in a more reliable way.
The main obstacle where I ran into this was swift-nio-extras, but I know many other libraries also link against zlib like swift-zip-archive and ZIPFoundation. It would be nice if we could have a solution that works reliably across more platforms including Windows.
I’ve done some exploratory work on zlib (RFC 1950) support in Swift with the swift-rfc-1950 repository. It currently provides basic zlib wrapping and Adler-32 handling around DEFLATE, though it’s not yet complete and will need improvements for production use. However, it does compile and build on macOS and linux, and CI shows it also builds on Windows.
If it would be helpful, I can invest effort to improve it (platform support, testing, API surface) so it could serve as a more robust basis for cross-platform zlib support in the Swift ecosystem.
Note that there is support in build-script, which builds the toolchain, for compiling zlib: that was added because it is a dependency of curl, which FoundationNetworking is built on for certain platforms. Unfortunately, Evan added those build config files more than three years ago, and then they were never used for anything in the public OSS builds, with most Swift platforms now building or including those curl/libxml2/zlib libraries in other ways, and the Powershell script build.ps1 for Windows has its own zlib config too.
If you want to include zlib in some more accessible form in the toolchain, perhaps in collab with @coenttb's work, you could just dust those existing build scripts off and start using them more, as I plan to for Android.
While we're talking about zlib, let's not forget and leave behind zstd if there is going to be any kind of special work around this.
zstd is way more cpu-effiecient. The efficiency difference usually-is/can-be noticeable in practice as well.
What about swift-nio-extras just vendoring zlib in its own source code? There's plenty of precedent (like llhttp, boringssl, FreeBSD's SHA1, Swift Profile Recorder vendoring the ELF symboliser ...) where NIO just vendors the code. This solved so many issues -- so I think should probably just do that here. When we originally made the call to just use the system's zlib, IIRC the call was that it's anyway available "everywhere" and wouldn't need a lot of manual updates because the code's very stable. Also I guess we knew the vendoring door would always be open.
Let's just do it? I'm pretty sure the NIO team would welcome a contribution or even just an issue requesting that. @lukasa WDYT?
Adds a c_nio_extras_... or similar prefix to all functions that create symbols (such that we don't clash with the system zlib or others that define functions with that name)
SPM doesn't really support dynamic linking today, and the zlib vendoring would just propagate more static linking perpetuating the code bloat that we see. I understand that this is the traditional route for server side programming with Swift, but this is definitely not a great model IMO (and definitely not what Windows tends to recommend for applications at least).
I belive that encouraging more shared libraries is the better thing to do here, most of which can be achieved by improving SPM to both build libraries and integrate with other systems. Of course, this is a longer term thing, and perhaps in the mean time, a package similar to swift-toolchain-sqlite with zlib is a good compromise.
Getting involved late, but I think @compnerd is absolutely right that the best answer, the right answer, is to get SwiftPM and Swift together to become substantially more friendly to being able to locate and integrate with system dependencies. I say both SwiftPM and Swift because I think both tools offer challenges that make this kind of integration harder than it needs to be, and a holistic solution would involve changes to both.
I am also cognisant of the fact that this is a problem for people now, today, and that nobody has committed to making the wide ranging changes that Swift and SwiftPM would need to achieve sensible and deep integration with system dependencies. I don’t want us to get stuck letting the perfect be the enemy of the good enough.
Simply vendoring zlib falls into the category of “good enough”. I think that goes double if we’re willing to make that behaviour optional, gated by a trait, such that it can be enabled situationally. The downside of that optionality of course is that it adds another multiplicative factor into the state space of NIO-based programs, yet another configuration to test.
A related note is that I’ll need to consult some legal experts to confirm that there are no licensing difficulties around including zlib as a source dependency before any such PR can be merged.
Quickly, on the question of what it would take to achieve great system package support in the Swift ecosystem, I think we’d need the following:
A way to enable SwiftPM to check whether a package has already been installed via a system package manager. This is necessary in a world where we want to be able to distribute pre-compiled binaries for packages, for example if we want Linux distributions to be able to rely on Swift packages as part of their core OS components, but it’s convenient in all sorts of cases, including making it feasible to distribute pre-compiled copies of some Swift packages on platforms with stable ABIs (e.g. via Homebrew).
A clear supported path to being able to import non-modularised package headers.
SwiftPM has decent support via the systemLibrary target type for importing C libraries from the platform, but it awkwardly basically makes you write the umbrella header yourself. This is a) essentially undocumented, and b) forces your library to evolve in lockstep with the library you’re using, making it quite painful to support rapidly changing libraries whose list of header files changes over time.
I can think of lots of ways to make this easier, with changes at different levels. We could change clang modulemaps to let you specify an umbrella directory without using an absolute path, searching the regular header search path for that directory. Or we could enable a way for build plugins to generate modulemaps on the fly. Or we could have a small DSL to express certain patterns for modulemap building. Or we could remove the requirement on clang modulemaps altogether. Or SwiftPM could transparently generate modulemaps from package-config files.
Regardless of the path we take, it is clearly not tenable to expect upstreams to modularise their imports when we’re the only ecosystem that expects them to do so.
Improvements to SwiftPM result packaging such that SwiftPM can produce more complex distributable artifacts. This would enable SwiftPM to lean more heavily on dynamic libraries while making sure that a single distributable unit is available.
I think this is a good start, but I’m sure there are more.
I agree with all the points Cory raised above. I think all three of these things are on the list of topics I’m trying to cover. I haven’t given much thought about systemLibrary other than they do need to be more powerful. Having actual knowledge of the system package managers does line up with the need to leverage external package ecosystems, be they source like vcpkg or binary as managed by the system package managers.
“non-modularized” headers is definitely on the list. This is a key requirement for embedded platforms as well. We should be able to adopt traditional C/C++ binary and source distributions and integrate them at the package manifest level with annotations to help integrate them into the build.
And extending the types of artifacts SwiftPM can create is also a key requirement for lots of things, be it firmware for embedded devices, container images, installers. make install has been a thing for a long time. SwiftPM needs to get there too.
These are all things on the table. It’s clear we need to open up the floor for discussion on the next steps for SwiftPM and build some momentum in the community to get these things done. I’m certainly seeing the passion for making things better and that’s a great start.