RFC: Moving SwiftNIO SSL to BoringSSL

The SwiftNIO team would like to ask the wider community to discuss a proposal to fundamentally change the TLS implementation used by swift-nio-ssl. This change would be scheduled to land in the next major release, which will also drop Swift 4 support (see Johannes’ proposal for more details).

The outline of the proposed change, as well as the reasoning and outcomes, is below. Please take a read and let us know your thoughts in the comments.

Proposal

We propose to change swift-nio-ssl to stop linking against the system copy of libssl, and instead to provide a vendored copy of BoringSSL. This change would come with a number of subtle runtime behavioural changes, as well as a number of substantially more disruptive changes around application distribution and OS behaviour.

The proposed change would drop support for OpenSSL and LibreSSL. The reasoning for this choice is discussed later in the proposal, and is not necessarily mandatory.

Note that for Apple platforms the recommended TLS solution will still be to use swift-nio-transport-services.

Motivation

One of the major difficulties encountered by server side software developers is that of distributing your built application. Most programs end up developing an implicit “build environment” that requires collaboration with the target deployment operating system to successfully configure.

Development

This integration with the operating system presents developers with a burden, both during development, and during deployment. In development, developers need to set up what amounts to a clone of their deployed environment. This is made particularly difficult in Swift on Server because the most popular development environment is macOS, but the most popular deployment environment is Linux. Most developers are not able to perfectly homogenise their development environment with their build and deployment environment, which opens up the door to problems that only manifest on build or deployment.

One of the major difficulties here has come from libssl. Swift’s supported platforms cover multiple versions of the various Apple platforms, and 3 versions of Ubuntu. Each Ubuntu version ships with a different libssl API by default (OpenSSL 1.0.1 for Ubuntu 14.04, OpenSSL 1.0.2 for Ubuntu 16.04, and OpenSSL 1.1.0 for Ubuntu 18.04).

Apple platforms provide no libssl at all, requiring developers who wish to replicate their Linux build environment to go and obtain a copy of libssl. Frustratingly it is very hard for developers to obtain the exact libssl version that their deployment target will use. For example, Homebrew currently makes available (correct at the time of writing) OpenSSL 1.0.2p, OpenSSL 1.1.1, and LibreSSL 2.8.2, none of which corresponds to the stable libssl available on the target Linux platforms.

Unfortunately, the absence of libssl on Apple platforms requires developers that want to ship applications that build on both Linux and Apple platforms to either require that their users read an installation guide to obtain their dependencies, or to conditionally depend on both swift-nio-ssl and swift-nio-transport-services. This conditional dependency imposes particular requirements on the way projects use Swift Package Manager, and it locks out users from being able to use libssl-based TLS on Apple platforms even when they may want to.

Deployment

The story for deployment is also tricky when it comes to libssl. This is because the produced application binary depends on having a copy of libssl available to it that is ABI-compatible with the one that it was built against. Unlike glibc, which aggressively remains ABI compatible, there are multiple incompatible libssl ABIs. The three major ones that affect Linux developers are LibreSSL’s, OpenSSL 1.0's, and OpenSSL 1.1's.

The result of this ABI compatibility concern is that a built binary can only run on a subset of Linux platforms: specifically, those that have a libssl that shares the same ABI as the one on the build machine.

Other popular languages in the server-side programming space, such as Go, have a compelling deployment story that simply involves copying a binary over to a new machine and running it. While this story is difficult for Swift to achieve exactly (due to our reliance on glibc), it is possible for Swift to get very close. However, any Swift application that links libssl is immediately ineligible for this kind of deployment model, as it will be indefinitely tied to this ABI constraint. Applications built on Ubuntu 14.04 against OpenSSL will run on Ubuntu 16.04, but not Ubuntu 18.04. This is an unfortunate, and unnecessary, limitation.

While we are actively using libraries like libssl widely in the Swift Server space, we will never have anywhere near as compelling a deployment story as languages like Go.

Cross-Compilation

When it’s necessary to link against platform libraries, cross-compilation gets increasingly difficult. This is because your cross-compilation toolchain needs to have copies of all of the system library header files and binaries for the libraries against which you intend to link. This incurs all of the difficulty mentioned above about obtaining the specific matching copy of your target libssl, with the added difficulty of requiring that you ensure it is built for the correct platform.

Feature Use

An additional limitation of the widespread use of the system libssl is that several popular systems have relatively old copies of OpenSSL that lack important features. For example, none of the OpenSSL versions in the supported Linux distributions support TLS 1.3, and due to the maintenance process of the supported distributions, they never will. Additionally, the OpenSSL version in Ubuntu 14.04 does not support ALPN, meaning that applications on Ubuntu 14.04 will not be able to use HTTP/2. Looking forward to new protocols, as QUIC (the transport protocol for the forthcoming HTTP/3) uses the TLS 1.3 handshake protocol, the absence of TLS 1.3 support in any of the supported Linux distributions mean that applications using the system libssl on those platforms will never support QUIC.

While users can avoid this by bringing their own, non-system copy of libssl, this presents its own challenge. Because the entire application is required to link one and only one copy of libssl, the entire ecosystem must be capable of supporting that version. This limitation has already affected Swift Server applications on Ubuntu 18.04, as many different libraries have had to make source code changes for OpenSSL 1.1, and not all server libraries and frameworks have completed this change. Until all of those libraries have adopted the new API, using new OpenSSLs is not possible. Naturally, there is minimal incentive for much of the ecosystem to adopt new APIs that are not supported by any of the system libssl versions, which makes conditionally using newer copies very difficult.

As a final note, this new copy of libssl would become part of the deployment artifact for the application, further complicating the deployment story.

Other Languages

Some other languages, such as Go, attempt to avoid this difficulty by using self-contained TLS implementations that do not depend on the operating system.

The engineering cost of writing a pure-Swift crypto implementation is high, however. Rather than boil the ocean by saying that we should do this, it would be better to begin by hiding the underlying C implementation from users. This would open the door to replacing the core implementation with one written in Swift, rather than in C/C++, without further disruption. As Swift Package Manager is capable of building C and C++ code, this should be reasonably straightforward.

Note that, for performance reasons, almost all cryptography libraries require the use of accelerated helpers written in assembly language. This is true for BoringSSL as well. This proposal is therefore gated behind adding support for building assembly code to Swift Package Manager. Code for this has already landed in SwiftPM master, and assuming nothing surprising occurs should ship with Swift 5.

Implementation

Currently swift-nio-ssl depends on swift-system-openssl, which causes Swift Package Manager to emit linker flags to whatever version of libssl can be found via pkg-config. This implicitly causes us to depend on the system libssl. We propose to replace this dependency with a new one, CNIOBoringSSL, which will contain a vendored copy of the source code of BoringSSL.

Opportunity

It has only recently become possible to make this change. As we need to perform symbol mangling to achieve this safely (see below), we have needed support from the BoringSSL project for this symbol mangling. This support was only added to BoringSSL on the 26th of August this year (in commit 8c7c635).

Packaging

This new dependency will not be its own package: instead, it will be a target within swift-nio-ssl. The use of BoringSSL is intended as an implementation detail for swift-nio-ssl, and will not affect the rest of the package ecosystem.

This is because BoringSSL does not provide any API or ABI stability guarantees. They are free to, and indeed do, break APIs regularly. If BoringSSL were exposed as as Swift Package Manager package, it would bump its major version number every release. This would make it impossible for the wider community to depend on: different packages would be pinning to different major versions, and dependency resolution would immediately fail.

swift-nio-ssl will present an API that does not contain any part of BoringSSL. If users need to use libssl or libcrypto themselves, they will need to bring their own copy of libssl.

Symbols

A concern of this approach is symbol clash. We will mangle the BoringSSL symbols to ensure they do not clash with any other package in the Swift ecosystem.

This concern is not theoretical: grpc-swift has already had to move away from BoringSSL once before due to symbol clash with the system libssl.

The problem here is that a Linux binary will only ever have one implementation of a given symbol name. If the binary is both dynamically linked against libssl, and uses a vendored copy of BoringSSL which has symbols that clash with the names in libssl, the portion of the binary that wanted the dynamically linked copy of libssl will instead get the BoringSSL implementation. This usually leads to memory corruption and eventual crashes, and is to be avoided.

The way we will avoid this is by mangling the symbol names of the BoringSSL symbols. The script that updates the vendored copy of BoringSSL will run a series of build steps that have the effect of prefixing all BoringSSL symbols vendored into swift-nio-ssl with a specific string unique to swift-nio-ssl. This will ensure that swift-nio-ssl does not bring symbols that clash with other copies of libssl.

As an example of this mangling, consider the libssl symbol SSL_new. In swift-nio-ssl, this symbol will instead be CNIOBoringSSL_SSL_new. As the string “CNIOBoringSSL” is entirely within the implicit NIO package namespace, there is no risk of clash with other libssl providers.

Other libraries in the ecosystem that may choose to vendor BoringSSL should do the same: this will ensure that multiple copies may inhabit the same program without conflict.

Updates

BoringSSL updates regularly, frequently dropping older, less-secure cipher suites and protocol versions. We will conceal these changes when they are non-functional, and follow semantic versioning when they are, thus hiding this rate of change from our users. We will provide all updates via Swift Package Manager.

One immediate effect would be that swift-nio-ssl will drop support for SSLv2 and SSLv3. The loss of these two protocol versions is highly unlikely to cause anyone any loss of sleep. Newer versions of OpenSSL have completely removed SSLv2, and SSLv3 is disabled across the vast majority of the web.

In the future, swift-nio-ssl will bring both new features and feature removals to all platforms simultaneously, rather than gating them on the specific operating system and OS version being used. This is a common deployment model: Chrome uses BoringSSL for this exact use-case, and many other tools and languages (such as Firefox and Go) ship their own TLS implementations they completely control.

Security Updates

Security updates will no longer be distributed by the OS package manager: instead, swift-nio-ssl will be carrying all TLS security updates, and users will need to update swift-nio-ssl to get them.

This is not a big difference from today. SwiftNIO and its ecosystem are a part of your application and can have security vulnerabilities of their own, and are not distributed by the operating system. As a result, it is already necessary to remain aware of the security content of Swift Package Manager package updates.

The SwiftNIO team has substantial experience managing security updates, both for SwiftNIO itself and for other projects. We have a system in place for deploying security updates, and will continue to use it for swift-nio-ssl when we’re distributing BoringSSL.

System Integration

The SwiftNIO team will enhance swift-nio-ssl to ensure that it continues to discover the system-provided certificate store, where this is available.

One of the downsides of using BoringSSL is that it is no longer aware of the default certificate store on most Linux distributions. Fortunately, this procedure can be dealt with manually by searching for the CA bundle at startup. As an example, curl contains a list of possible locations that covers most Linux distributions (curl/CMakeLists.txt at f859b05c6686b2c5a41fd7805f164229b3c6d7c8 · curl/curl · GitHub).

Swift-nio-ssl will include this logic by default, so this should not result in a noticeable regression for any users.

Version Numbers

One concern about using BoringSSL is that it has no notion of a stable version number. This absence of a stable version number can make it difficult to be confident about exactly what version of BoringSSL is in use when attempting to investigate issues with your application, or when investigating if you need an update to obtain a BoringSSL bug fix or security update.

To combat this, swift-nio-ssl will include a method that can be called to obtain the SHA of the git commit used to obtain the BoringSSL sources. This commit will be from the upstream repository, at https://boringssl.googlesource.com. This SHA will also be present in a code comment in the swift-nio-ssl Package.swift file.

Result

The immediate result of this change will be that swift-nio-ssl gets easier to build for all users on all platforms. A large number of subtle compiler errors will no longer be encountered by users, particularly those new to the ecosystem who are attempting to start developing on macOS.

Additionally, it will become that little bit easier to build static binaries for Swift Server applications, and so to distribute Swift Server applications more easily. In particular, they will no longer be tied to the presence or absence of a libssl on the platform: they’ll bring their own along with them when they need it.

The gap between development and production environments will get smaller, as both environments will be running the same code using the same libraries.

Finally, we will improve the security and functionality of the entire Swift Server ecosystem, regardless of the operating system they are targeting. This will help keep more of our users more secure, as well as more up-to-date with the modern web.

23 Likes

Thanks for the writeup @lukasa. I have comments but want to clarify something first:

There are all sorts of reasons why a server-side application might want/need to use OpenSSL functionality, not just TLS. If NIO goes ahead with this plan, but hides the BoringSSL, then all the problems you outline will still impact a very large number of server-side applications. This seems like it might solve a problem for NIO, but not a problem for developers.

Have you considered a variation of your plan where BoringSSL is made available as a package?

Yes. The proposal above discusses this, but I'll reiterate here in more detail.

BoringSSL has no stable API, none whatsoever. This means that the only way to depend on its API is to pin to a specific version. As you point out, something like this is likely to be widely used, meaning that many different parts of the ecosystem will start to depend on BoringSSL. It is inevitable that they will all depend on different versions. The moment they do that, it becomes extremely hard to build any non-trivial package with dependencies (and I'd say it becomes effectively impossible in the absence of a SAT solver for SwiftPM, something I am not proposing to build).

The only other thing that could be done would to commit to shipping BoringSSL entirely wrapped in Swift APIs. While the SwiftNIO team just about has the bandwidth to maintain such a thing for libssl, we absolutely do not have the resources to do so for libcrypto. Most of swift-nio-ssl's internal code is extremely thin wrappers around the libssl APIs, meaning that its APIs are unsafe and distinctly non-Swifty. The burden of solving everyone's problems with crypto and TLS is a big task, and something the community needs to handle together. Sadly, it's an ocean I cannot just boil myself.

In the absence of the perfect solution, we should not let the perfect be the enemy of the good. In this case, we gain a number of advantages, as well as provide the infrastructure to allow the rest of the Swift ecosystem to follow this path if they choose to.

1 Like

Everything you say makes sense, but I don't see how it all plays out. What is your guess as to what % of users of NIO do not also need some other OpenSSL functions? We might have some infrastructure that could help gather a number.

I'm not arguing we shouldn't do what you propose, it overall seems reasonable and well thought out to me, but I'm not convinced it will (on its own) solve very much of the declared problem.

Certainly I understand the NIO project cannot necessarily sign up to vend stable Swift wrappers for the BoringSSL APIs. However, suppose your infrastructure is awesome and then every web framework adopts it and also vendors BoringSSL privately? That doesn't seem like a good solution either.

If your vendored BoringSSL was a separate package, with a stable but minimal API for only what NIO needs, then it at least opens the door to other projects extending that API to accommodate additional needs), right?

It depends on what constitutes "other OpenSSL functions". If we say any crypto or TLS, regardless of whether that actually ends up using OpenSSL, I'd say it's fairly high. Right now it's probably upwards of 95% of users, due to the preponderance of web frameworks in the Swift ecosystem and the use of URLSession.

It does. It raises some thorny questions about how we handle API design, and indeed what the design goals of such a thing should be. My view here is that we can always evolve to that, but that I'm a bit reluctant to propose to do it straight out of the gate. It's much harder to put that genie back in the bottle than to take it out later.

I would like to give a +1 for every hour that I believe to have spent in discussions around SSL issues in the Swift on Server ecosystem with nothing much coming out of it. Depending on the platform's libssl just doesn't work but is also kind of the only way (apart from vendoring BoringSSL). So far vendoring didn't seem possible until BoringSSL added symbol prefixing.

This is the first really workable solution and therefore + :100: (hours I would like to have back in my life :wink:) , getting this going would make everything so much better :slight_smile:

3 Likes

Is there a goal to provide a Swift-based implementation of Crypto and/or TLS long term? Or is that something you're hoping the community can provide via the SSWG incubation process?

This is probably highly desirable but a significant amount of work!

It's something I'd love to see! Right now, however, it's not anywhere on the SwiftNIO roadmap. That's not to say we'd never choose to do it, but as things stand today I can confidently say that the SwiftNIO team has no plan to work on such a thing.

As an academic discussion I'll note that writing crypto implementations in pure-Swift is...probably not advisable. In particular, it is very hard to confidently write code that is side-channel resistant in Swift: the compiler just doesn't get you close enough to the hardware. There'll likely always be some C code buried in there. That means the highest-value target for a crypto library is great, hard to misuse, Swifty APIs that ultimately delegate to C-based crypto primitives. Such a library could then be the basis of a pure-Swift implementation of the TLS protocol itself, which is something that can happily be written in pure Swift.

However, right now there are enough good C implementations available to us that we can successfully move forward on more important protocol work (HTTP/2 now, and we are looking to start work on QUIC in the new year). TLS is just nowhere near our biggest limiting factor.

3 Likes

this is something that came up before in the context of SSWG and we should totally continue this discussion there. given the issues @lukasa mentions above, this may not be a pure swift implementation, but the need for a solid crypto library for server applications is real and beyond the scope of what SwiftNIO can provide

A huge +1 from me. Thanks a lot for all your effort here.

(I am the manager of BoringSSL work at Google.)

To reiterate something from BoringSSL’s homepage: “Although BoringSSL is an open source project, it is not intended for general use, as OpenSSL is. We don't recommend that third parties depend upon it.”

What that means is that, if Google products no longer need feature $x, we may remove it. Similarly, new feature work is significantly driven by Google’s long- and short-term goals. You might consider that Google’s compatibility needs are very wide (which is true) and that Swift users are going to be a subset of that, and that Google’s feature desires are likely to be similar to everyone else’s (plausible) and so the mismatch doesn’t matter. But BoringSSL doesn’t have the kind of community involvement that you might be used to from other open-source projects.

The symbol-prefixing, for example, was added in order to support a specific Google need. While it should also work for you, we’re still working on making it work on Windows for one thing.

If you do use BoringSSL, vendoring and prefixing would certainly be necessary. We would also strongly recommend keeping up to date with the latest version. Major Google products, for example, operate on a weekly import cycle. This is important because rapid feedback about issues allow us to respond (although see above), while feedback about code that was landed months ago is generally too late. (Keep in mind that Google is largely oriented around a monorepo where nearly everyone builds from the same versions: http://delivery.acm.org/10.1145/2860000/2854146/p78-potvin.pdf .)

If you want to wrap BoringSSL functions in a stable API, you might want to run it by us. While we might not have time to give detailed feedback, and don’t know about Swift, we might be able to usefully mention specific APIs that are not suitable or have superior replacements. (Hopefully such things would be noted in the headers too: BoringSSL - Headers .)

If you wanted to expose libcrypto functions too (e.g. SHA-256, AES-GCM) we could also advise about that. For both TLS and general crypto, we’re biased, but the Go APIs (e.g. tls package - crypto/tls - Go Packages) are a decent starting point. There are things that I would change with hindsight, but they’ve done reasonably well over the years.

12 Likes

Thanks Adam. For the moment the concrete plan remains to avoid exposing public APIs that concretely map to any specific BoringSSL feature, instead to only broadly use it to provide a TLS implementation for SwiftNIO.

As to version updates, we’ll be expecting to keep up a regular release schedule, though we expect them to have relatively little functional impact for most users.

1 Like

I agree that having a consistent TLS implementation across Linux versions is a big win, especially when it comes with TLS 1.3 etc. But there are downsides to this too.

Shipping a vendored copy of BoringSSL as part of swift-nio-ssl will increase the compiled binary size of applications. Have you measured how big the increase is?

As others have commented, this does not solve the problems we have with crypto. As most server-side Swift application will continue to link OpenSSL for things like HMAC, AES etc (needed for JOSE) this will also increase the runtime memory footprint of applications too, as we'll now have both BoringSSL and OpenSSL loaded. Have you measured how big that increase is?

I am interested in exploring @ddunbar's suggestion of a separate package. While it would let the genie out of the bottle, long term it seems inevitable?

With respect I think it is. You can get a fixed system libssl by running apt-get whereas in this new model users will be required to recompile their applications. Immutable infrastructure is a thing, but even so, the difference is not insignificant.

OpenSSL is an ongoing rich source of CVEs and many of these apply to BoringSSL too. Is the SwiftNIO team committing to shipping fixes in a timely manner and what detail can you provide about the systems you have in place to enable this?

As an academic discussion I'll note that writing crypto implementations in pure-Swift is...probably not advisable. In particular, it is very hard to confidently write code that is side-channel resistant in Swift: the compiler just doesn't get you close enough to the hardware.

Paging @krzyzanowskim ...

1 Like

Not yet, I'll get you some sample numbers tomorrow.

Again, not yet, will get you some sample numbers tomorrow.

I think the fact that it seems inevitable is only because we're not talking in specificities. :smile: What specific thing seems inevitable to you?

If we take the most specific thing, a SwiftNIO-maintained Swift-wrapper around BoringSSL for general purpose TLS and crypto usage, I don't consider such a thing an inevitability. In particular there have been concerns raised in this thread about the broad usage of BoringSSL for this purpose, and I share them: the only reason I am comfortable with this proposal as it stands today is how constrained the use of BoringSSL is.

The much more nebulous notion of a cross-platform TLS and crypto library is obviously much closer to an inevitability, but it seems reasonable to ask the question as to whether such a thing already exists. In fact you could make the case for there being more than one already, none of which are currently maintained by the SwiftNIO team.

I raise this to say that while I agree that there is inevitably going to be at least one cross platform TLS and crypto library, I very much do not agree that there is inevitably going to be a good, SwiftNIO-maintained, cross-platform TLS and crypto wrapper around BoringSSL. More importantly, I think the idea is not really a blocker for this proposal.

As I have said above, I'm very interested in the idea of a cross-platform TLS and crypto library. What I'm worried about is letting the perfect be the enemy of the good. We can always move SwiftNIO over to such a library when it comes into being: this proposal does not limit our ability to do that in any way. Adopting this proposal does not prevent us building the library you and @ddunbar are envisioning.

As noted above, SwiftNIO is already a security-critical piece of infrastructure. It has already shipped a release for one CVE in its short life, and with all the will in the world there is good reason to assume there will be more in its future. Users should not be being encouraged to assume that apt-get update && apt-get upgrade -y will solve all their security problems.

Yes we are. As part of this proposal we would be asking the BoringSSL team to proactively notify the SwiftNIO team when security fixes are landing. We would be preparing and releasing updates to coincide with or very shortly follow any BoringSSL change that needs to be shipped. Where possible these patches will be shipped for all extant minor release versions of swift-nio-ssl.

In addition, we have a preexisting procedure internally for using Apple's security fix reporting infrastructure to issue CVE numbers and ship updates for SwiftNIO code directly. This procedure would be used for any vulnerabilities in swift-nio-ssl itself.

2 Likes

As requested, I've done some digging. I have built swift-nio-ssl's NIOTLSServer sample binary against the current master branch, and against a BoringSSL-based branch. This build was done on macOS, but it is likely to be equivalent on Linux. I used release mode, as this is the relevant compile mode for application distribution.

The master branch binary size is 2272668 bytes (~2.2 MB). The size of the BoringSSL-including version is 4196868 bytes (~4.0 MB). This implies that the move to BoringSSL will increase compiled binary size by 1.8 MB.

That number in the abstract is not useful, so I quickly built the "hello world" applications with Kitura, Perfect, and Vapor for perspective. When built in release mode the binary size of those applications is:

  • 4.8MB for Kitura (including the 5 dylibs it links)
  • 6.6MB for Vapor with Leaf (it compiles statically, no need to chase its deps)
  • 4.2MB for Perfect (also compiles statically) (as a side note, Perfect already ships a statically linked copy of OpenSSL for its crypto purposes)

This implies a worst-case binary size penalty of 40%. Certainly worth considering in the trade-offs.

Getting good numbers here is somewhat non-trivial (I bumped into build issues with both Vapor and Kitura when using a SwiftPM new enough to have support for compiling assembly code), so I whipped up a quick NIO application that uses BlueCrypto to hash the data sent to it and return the digest. Not very interesting, but fine for this.

Using swift-nio-ssl@1.3.2 the total memory usage (as returned by pmap) was 237196K. Using the BoringSSL variant, the total memory usage again returned by pmap was 238848K. The delta there is 1652K, or about 1.6 MB, approximately equivalent to the increased size of the binary. I consider this to be the "expected" result.

2 Likes

For those who don't like doing math I'll quickly note that the 1.6MB increase here is a 0.6% increase in the memory footprint of the application, though this is total usage including all mapped dylibs. If people would like a different memory usage number, please shout.

1 Like

Thanks @lukasa for the effort of providing the hard numbers! I guess that shows that binary size & memory overhead won’t be issues with this solution. On the server-side where that is almost irrelevant anyway but under 2 MB wouldn’t even be an issue for mobile (mostly thinking about Android as on iOS we’ve got NIOTransportServices which uses Network.framework for TLS anyway) or Raspberry Pie etc.

Why was BoringSSL chosen for this instead of LibreSSL or OpenSSL? What makes Google's project better suited for this purpose than those two, especially considering their project page is all but begging others not to use it?

Perhaps this was explained in the OP, but I couldn't find it after one read-through and a couple more skims.

2 Likes

It wasn't explicitly called out, but it's covered implicitly in a few points above.

The primary technical reason is that neither project has any support for namespacing or prefixing their symbols. This means vendoring them into the codebase requires either substantial manual rewrites of the codebase when including it to manually mangle all symbols, or running into problems with symbol clash whenever someone uses the system OpenSSL/LibreSSL. Symbol clash is a complete nonstarter, so mangling would be the only option, and so far as I am aware there is no good technical solution to symbol mangling that does not require active cooperation of the project involved. BoringSSL is doing this: the other two do not.

The other reason is feature development. In general, BoringSSL is furthest ahead in feature development, followed by OpenSSL, trailed at quite some distance by LibreSSL. In this case, I am particularly concerned about unlocking the ability to build and ship implementations of newer network protocols, particularly focussing on QUIC. OpenSSL right now does not provide access to the hooks needed to perform the QUIC handshake, and has no plan to do so for the foreseeable future. BoringSSL does.

Don't consider the argument about QUIC to be a specific argument, consider it an example of a category of argument. Future network protocols will likely also require specific things from the TLS stack, and waiting for OpenSSL to build and ship them greatly slows our feature development velocity.

4 Likes

The main reason this can be made to work now is that BoringSSL now (since about two months ago) supports symbol-prefixing which means that we can ship symbols that for sure won't clash with anybody else's. OpenSSL & LibreSSL don't support this. @lukasa will be able to expand if there's other reasons but this one alone is really enough.

Example: Every libssl (Open/Boring/LibreSSL) will have an SSL_write function. That means if another shared object calls SSL_write we should be in big trouble if we had multiple libssl's in one process as it's undefined which SSL_write version will be run. But different libssl versions have different data-structures. So it's very possible that you're using the SSL_write function from one libssl with the datastructures from another libssl and indeed has hit the grpc-swift project before. The result is totally undefined, might be weird crashes, might be random memory corruption, might be okay, might be anything really.
Long story short: You can never (on Linux) mix different libssl versions in one address space, unless you make the symbols not clash and that's exactly what the symbol-prefixing is: NIO's version will just be called cnioboringssl_SSL_write and we won't clash with anybody else :slight_smile:.

2 Likes