Our journey with Swift thus far - some notes and reflections

Community Showcase
1

Hello!

Background

Our team is quietly working on developing a new electronic trading system using Swift on macOS and Linux, building on our experience from two previous projects that have been done in C, Objective-C, C++, Java, C# and even for a while, homegrown DSL. All running on multiple OS:s at various points; NeXTSTEP, OpenStep, Windows, HP-UX, Solaris, Linux. These past projects have been fairly sizeable, with over 25 million lines of code and several hundred components between them over a period of over 25 years with teams ranging between 40 and to 100+ engineers over time.

These are very high performance, distributed cluster systems on the server side, combined with sophisticated frontends with visualisation and monitoring. The server side provides APIs for customers to hook in custom plugins that perform e.g. trading activities or pricing, which need to run in-process typically (i.e. dynamically loaded). This has been done with both C++, Java, C, and custom DSL. When we talk about "server side" here, it is in the more traditional sense with high performance IPC etc, so no HTTP, JSON or similar in general.

Delving into Swift's ecosystem has provided us with valuable insights into both its advantages and areas where there's room for growth. The feedback we're sharing is based on our journey so far, and the hurdles we've encountered, with the intention of offering constructive thoughts for Swift's continued refinement. There's still some way to go before we're launching properly, but we wanted to take this time to share a few of our reflections and musings.

Why Swift?

We have a vision that our next-generation system would not only use an expressive, memory safe and highly performant language, but also that we should be able to use a single technology stack from top-to-bottom. That enables great code re-use, engineers can more easily pick up work in various layer of the stack and we can benefit from the unified implementation of various types between both client and server pieces more easily. We also have a commercial necessity to be able to deploy on Linux on the server side in addition to macOS for visualisation (and possibly other Apple platforms going forward).

Swift ticks most of those boxes while also providing the ability to create native, good user interfaces using SwiftUI. There are a few other options, but for various reasons they didn't meet our criteria (and not relevant to this post).

Good stuff

Overall design

  • Swift's design of being "safe by default" - the strict type checking felt a bit frustrating at the beginning (ok, still sometimes ;-) but after coming over the initial (small) hurdle, we could really see that it paid off handsomely - code much more often will work immediately as expected (especially if compared with C-style languages) after going through the compiler. Major refactoring work is a lot faster and more robust with Swift in general, something that has significant value to us. The number of runtime errors is just a fraction of what's common with more traditional tooling. We're really happy with that aspect. The static data-race safety guarantees coming up is just the icing on the cake.

  • The language expressiveness is overall very good and we think the overall philosophy has a set of good tradeoffs. We're a little bit worried that the complexity of the language continues to grow for newcomers, but hope the philosophy of progressive disclosure will make it manageable over time.

  • The pace of development of major language features during the last few years has been great, with several things being right up our alley and being really useful to us, e.g. Structured concurrency/Async+Await/Actors, Distributed Actors, Memory ownership (super important and great it is being worked on!), and overall improvements to generics and Macros.

  • The open evolution process is overall really good (modulo some hiccups like result builders). To have visibility in the roadmap really helps and is appreciated - e.g. we saw that Distributed Actors and the new Predicate type was in the pipeline, and could choose to just stub out trivial implementations internally and then move over as they shipped - very good.

  • ABI stability is really a key feature - we do want to use library-evolution for our customer-facing SDK:s and this is just a great feature - not just for Apple as a system provider, but also for us as a third party. We need to be able to ship shared libraries that customers can link against so we can load plugins dynamically with resiliency and type compatbility.

Ecosystem

  • The ecosystem of packages usable for server-side development was more extensive than expected - we've over time also have helped contribute to some of the pieces that we've missed, like e.g. HDR Histogram, swift-kafka-client and performance benchmarking

  • We also are very happy with the native rewrite of Foundation (and are starting to use certain new features like Predicates heavily already) and are really looking forward to its integration as the built-in implementation for Linux - it moved us from having a no-Foundation policy to now instead adopt it.

  • SwiftPM plugins (and Macros) are enabling a wider range of workflows.

In general we are very happy with the language itself and see significant progress over time.

Major challenges

Much of issues are centered around richer dynamic library support, API packaging and Linux platform support, something we need for packaging products properly.

  • Dynamic Library support on Linux with library evolution - even a limited officially supported package with clear limitations would be useful (as our customers will have a completely controlled environment, it's possible to mandate specific library versions and patch levels for OS etc).

  • Library support for artifact bundles (we are currently running with a custom toolchain that supports xcframeworks on Linux to work around it, but that's not long term sustainable).

  • Concurrency runtime analytics tools could be richer, especially for Tasks - while we have some of the tooling in Instruments, it would be fantastic to also have support for accessing similar task information (async task stack backtraces etc, for creation/running/sleeping) in LLDB (especially import on Linux that lacks Instruments...), including inspecting Task locals.

  • Compile and runtime availability checking support for third party APIs is not available currently.

  • Fundamental support for dynamic libraries overall as a first class citizen, e.g. targets can't be linked dynamically .

Minor challenges

Concurrency and testing

Tooling and platform support

  • Linux support has been lagging at times (e.g. Macro support), which has postponed adoption of newer releases for us and it hasn't always been transparent .

  • Build times on Linux when using the new Foundation are quite painful (we need to not only build swift-foundation, but also swift-syntax due to use of macros - it's a well-known problem of course, but just wanted to mention it as it causes long CI turnaround for us).

  • Build Stability and Performance - we very often need to nuke .build or SwiftPM caches or Xcodes derived data due to weird issues (like missing Packages in Xcode), e.g. things seem to rebuild more than expected (e.g. this example for Benchmark).

  • SwiftPMs model with products/targets is a bit problematic for larger projects; targets are exposed to downstream projects, targets can't depend products in the same package, target name conflicts (aliasing doesn't always work out there) with module names (perhaps not SwiftPM exactly).

  • Local development of dependencies is a bit clunky, Swift package edit only solves this partially. We end up with a sometimes convoluted and non-ideal project structure with some tools / workaround on top of it.

  • LLDB support has been spotty with many crashes on Linux especially (and little updates on issues reported), it is improving over time though.

  • Libdispatch as the underpinning for thread management (on Linux) continues to lag the macOS implementation (assuming primarily due to the difficulty of merging newer implementations as libdispatch is heavily using Mach:isms). Perhaps the approach to integration with thread management could/should be reconsidered at some point, but understandably there are a lot of historical considerations to take here as there's a lot of code using Dispatch.

Performance

  • Performance issues are sometimes a bit hard to diagnose and address (e.g. unnecessary copying, we can get tons of metadata lookups when using existensial/generics if not getting the incantations right, ARC traffic, etc) - and need workarounds that allow the compiler to specialize properly. It feels like some sort of instrument for analyzing anti-patterns in Swift could help, would be happy to discuss details.

  • Generics, while having improved over time, still is not without friction and it is fairly complex at times to get the desired behavior. From a performance point of view it's a bit of hit-or-miss. It's challenging to guarantee that things will get specialised, and it is easy to make a nice generic refactoring that just runs over a performance cliff that you don't expect. It would be great with improvements in this area, e.g. there have been discussions about various annotations (e.g. to guarantee specialization).

  • KeyPath performance is problematic - unfortunately the performance of KeyPaths has forced us to do workarounds even in fairly simple cases like sort comparators as they would take over samples completely - it'd be nice if they could be used without fear.

  • Deeper documentation and hints on best practices for performance when using generics and protocol oriented programming (related to the previous point). Basically a follow up to the older performance best practices, but with a bit of more focus on cross-module considerations.

  • Analyzing program behaviour is missing some runtime hooks to better understand runtime behaviour and perform better benchmarks.

Overall, we'd welcome further investment in runtime performance, compile times and overall robustness (SwiftPM / Toolchain / Linux support on par).

Closing notes

In conclusion, our experience with Swift, while largely positive due to its modern features and robust ecosystem, also highlights areas where enhancements could further refine and expand its capabilities, especially for slightly more complex and advanced development scenarios. We hope this feedback serves as a constructive contribution to the ongoing development and improvement of Swift, reflecting our commitment to help pushing the ecosystem in a positive direction. We'd be happy to elaborate and engage in discussion on any of the above topics.

Many thanks to everyone involved in making Swift with related packages available - we think it's on a great trajectory, looking forward to continue on our journey!

A special thanks to everyone who participates and shares their knowledge on these forums, it's an invaluable community resource, thanks!

Joakim

111 Likes
2

Thanks so much for sharing this @hassila! Getting feedback from large-scale systems helps tremendously in understanding what limitations you hit. Also your contributions to the ecosystem have been amazing over the years. Especially the benchmarking package has been tremendously impactful in our projects since it made writing benchmarks easy and we started writing them from the beginning!

11 Likes
3

Thanks for the writeup; You're really hitting all the painpoints while also mentioning some of the good stuff, warms my heart :face_holding_back_tears: It's great to hear about the painpoints from your adopter's point of view -- thank you for taking the time for this writeup.

It's been awesome to see you folks contribute to the ecosystem and adopt the latest and greatest stuff including distributed. Thanks again and I hope we can keep chipping away at those pain points listed!

I'd actually like to ask you to file an issue for the task and task locals inspecting things, I wanted to poke on this and put some of this into swift-inspect specifically :slight_smile:

Oh and good news about the "send", it is coming in this form: Task { isolated target }, and I'll think about what we should do for distributed remote calls to avoid reordering there as this isolated spelling won't work there (can't peel off the distributedness like that). Thanks for explicitly calling it out!

17 Likes
4

Join us in the swift-testing forum category if you have questions, comments, suggestions, or extremely relevant stand-up routines! :grimacing:

4 Likes
5

That's great news, I managed to miss that, thanks! Also hoping for the distributed part as a follow up, "oneway void" is useful at time...

1 Like
6

Will definitely do, I'll admit we haven't gotten around to moving any real stuff over yet, but hope to do a POC in the near future.

3 Likes
7

Build Stability and Performance - we very often need to nuke .build or SwiftPM caches or Xcodes derived data due to weird issues (like missing Packages in Xcode), e.g. things seem to rebuild more than expected

The same is true on macOS. I love Swift but working on a large codebase can be really painful.

13 Likes
8

Thanks for the detailed info (your lldb link is broken).

You have gotten involved in the OSS package ecosystem already: what would stop you from investing in the toolchain too? The Browser Company hired @compnerd and is regularly sending Windows pulls back upstream. I wish more companies using the toolchain intensively did the same, particularly for platforms where there is a pure OSS toolchain, like linux.

2 Likes
9

speaking on behalf of the company i lead (which is in a comparable situation), we would love to invest more in the toolchain, but we are simply stretched too thin and are unable to make all the commitments we would like to.

ultimately "investment" has to come from somewhere, and it has been challenging post-QT to persuade people sitting on large piles of cash to make these sorts of collective-interest investments instead of parking it at the fed.

TLDR; too few personnel, too high interest rates

4 Likes
10

Many thanks for sharing experience and more broadly for all the contributions...

I find refactoring a bit of a pain point (relative to eons past, when Java refactoring was robust and extensible in eclipse, and structure101 displayed layers and refactoring plans). If you (or anyone else) are using any helpful (non-obvious) tooling or resources, would you please share pointers?

On the challenge of tooling and platform support, it seems like interest is high but perhaps not engaged for lack of coordination. Do you have ideas for that, or ideas on who might have ideas?

1 Like
11

For me that happens with any non trivial project somewhat regularly. And with Macros, Swift Syntax and large git repos as dependencies cleaning DerivedData means a lot of lost time.

1 Like
12

Thanks for the friendly feedback @ktoso, I've done that at:

But also notes that swift-inspect currently isn't available for Linux (although there is a PR for x86/64, ARM support would also be missing):

Also, it seems all the concurrency support is Darwin only at this point, which I'd understand for threads, but I would have expected Tasks etc to be the same across platforms?

Also, I've somehow missed swift-inspect, but trying to run it against a binary is a no-go:

hassila@ice ~/g/o/frontend (main)> sudo swift-inspect dump-concurrency 14416
Password:
unable to get task for pid 14416: (os/kern) failure 0x5
Failed to create inspector for process id 14416

Any pointers on what the requirements are? Looking at the documentation at swift/tools/swift-inspect at main · apple/swift · GitHub it doesn't give much hints unfortunately if there are any other prerequisites.

2 Likes
13

Sorry, I don't have any hints on better tools, I agree the tooling there could be improved (Xcode is a bit flakey and often fails there for us) - here's hoping to an LLM-enabled magic refactor tool for the future... My comment was primarily around that the refactoring have been fairly quick and robust thanks to the type safety of the language, when we get it to compile and go through the test suites, it typically works very well.

1 Like
14

Fixed, thanks.

Currently mainly a matter of available resources, we don't have anything in principle against it and would be happy to contribute when we can (and have historically done so with e.g. financing platform support for Solaris for clang and helped port libdispatch to Solaris once upon a time at previous companies). Currently we're (again...) in a startup phase so a bit more limited visavi resources at this stage. That being said, we've done a little Linux-platform specific work too (added initial io_uring support to SwiftNIO and still working on a follow-up with outbound write support).

3 Likes
15

Thanks for the issues; getting swift-inspect to work would indeed be very good... Can't promise anything, but at least personally I've been interested in getting that done.

2 Likes
16

Thanks! Does swift-inspect work at all right now for anyone, and if so what’s the trick? (It fails out due to permissions on macOS even with sudo)

1 Like
17

This is a great writeup. I'm interested in Swift as a modern compiled cross platform language, and it seems like the bones are all there to make that happen. Being safe by default, the integrated documentation with DocC along with the ability to make nice tutorials are all great features. It seems like we just need a bit more momentum from people building and deploying things outside of the Apple ecosystem to entice more people ot give it a shot, but like I said, the foundation for Swift is good.

5 Likes
18

one thing i am curious about is how you managed to work around Swift’s lack of Amazon Linux 2023 support. we chose to build our stack on Amazon Linux, on the assumption that this would be the most cost-effective choice, since Amazon Linux is specially tuned for performance on AWS. but this ended up being a terribly costly mistake, as Swift is something of an outlier in its lack of Amazon Linux support and we expended a ton of lost time trying to bootstrap support ourselves.

  • what AMIs are you deploying to (if using AWS)?
  • what environment are you doing most of your development in? (Xcode, VSCode devcontainers, etc.)
3 Likes
19

We early on decided on Ubuntu as it was the primary development platform for Swift on Linux - really just to avoid as many issues as possible with platform support (burned from previous experience similar to yours now). Now, our deployments are typically privately hosted on dedicated server in various private colos, rather than cloud based, so while we at times use AWS for certain kinds of testing and development (and other internal infrastructure), it is really not our primary deployment environment. We typically pick up the current LTS release when Swift support becomes available, so currently at 22.04 and looking to update to 24.04 as soon as it's officially supported.

We primarily use Xcode for development (with a few engineers trying VScode, don't know what the current experience have been there though).

5 Likes
20

The Browser Company have recently shared their VSCode development setup (for Windows, though).

1 Like