A Few Take-Aways From the Rust Ecosystem

Evolution Pitches
, ,
1

In short, this is a few take-aways from my experience with the Rust ecosystem that we could adopt to make Swift even better:

  • A toolchain manager for Swift
  • A package registry
  • More "Swifty" testing
  • Generated documentation
  • An online playground
  • Command-line benchmarking
  • Make Swift more "self-contained" from Xcode
  • Reduce the size of the Swift toolchain (if possible)

Note that this isn't really a single proposal, but a number of discussion staters, each of which could be its own proposal.

Details

Recently, I've been learning Rust. It's a really great language and I can see why Swift pulled so much inspiration from it. For the most part, I think Swift just took the best parts of Rust and made them even better. Swift gains a lot of the safety of Rust, but with code that's easier to write, cleaner and more expressive. For the most part, I still prefer the experience of programming in Swift.

However, there is one area where Rust really shines over Swift: Rust has an amazing ecosystem. The experience of installing Rust, configuring it and getting projects started is a very cohesive, experience. Rust is IDE-agnostic, simple and straight-forward, and everything works well together. It's a self-contained, comprehensive suite of tools to write code.

Once setup and running, Swift shines. But, getting there is a bit clunkier. It's not hard, but it doesn't have that cohesive feel that Rust does. All of this isn't really an issue for traditional Swift programmers, who have primarily used Swift for iOS/Mac development and got it bundled with Xcode. However, we want to bring more types of Swift programmers to the table: those on other platforms, other IDEs, etc. As part of the "all are welcome" future of Swift, making the non-Xcode installation/setup process more friendly will reduce some of the barriers to getting new Swift adopters.

On Mac, Swift is very reliant on Xcode and has lots of ties to Xcode that feel a bit awkward. While Xcode is a great IDE and should be the best IDE for Swift development on Mac, Swift itself should feel more self-contained. On Linux, it is (because, of course, there is no Xcode on Linux). But this just makes the experience of installing Swift different on different platforms. To truly become a multi-purpose, multi-platform language, I really think we need Swift to be free of Xcode on all platforms. Xcode can embrace Swift and be an awesome Swift IDE, but Swift should be agnostic. A good example of this separation beginning to work is SourceKit, which Xcode uses, but allows other IDEs to provide the same experience as Xcode as well.

Another impediment to Swift is the size of the toolchain. The Rust toolchain is 117.5 MB. By contrast, the Swift 5.3 toolchain (Mac) is 1.61 GB. That's over 13 times larger. Why is it so much larger than the Rust toolchain? There may be a good reason, but it may be something to think about. The size makes Swift feel a bit heavy. Rust does a lot (compile code, manage packages, generate documentation, run tests, run benchmark, etc.) for a lot less. Can we go on a diet?

Here is a rundown of a few Rust ecosystem features I wanted to call out, along with some proposals about how we can adopt them in the Swift community:

Online playground

Rust: Rust has an online playground on its homepage.

Proposal

While Swift playgrounds are great, they only really work well in Xcode. Having an online playground right on swift.org would be a great way to get new Swift developers hooked before they even download the toolchain.

Installation process

Rust: Rust can be installed with rustup, a tool that manages your Rust toolchains and keeps Rust up-to-date.

Proposal

I'd propose we add an official Swift toolchain manager like rustup. Let's call it swiftup (probably not what we'd actually call it, but as a placeholder for now). Like rustup, downloading swiftup this would be the first step for any OS. It could be by running a curl command like Rust, or via a package manager (Homebrew, etc.). Doing so would automatically download the latest Swift toolchain. swiftup would also let you download/install alternative toolchains, keep Swift updated, allow toolchain switching (globally and per project), etc.

We actually already have a Swift tool that does most of this: swiftenv(swiftenv Homepage). swiftenv is great, but would be even better as an official part of Swift.

Many Mac users already get Swift by default with Xcode. While this would still appear to happen with swiftup, Xcode would no longer bundle a Swift toolchain. Instead, Xcode would ensure swiftup was installed and then use it to download/manage the preferred toolchain (downloading it to /Library/Developer/Toolchains vs. inside the Xcode package). Unlike today, setting your toolchain from within Xcode and from the command line would do the exact same thing. This would be a more unified approach, not just across platforms, but also on Mac.

Package registry

Rust: Rust has the cargo tool and the crates.io as a package registry.

Proposal

I know this one is already in the works for Swift, but still worth calling out. Having an official registry to search, install from and publish to right from the swift package command will be a big win. One thought: perhaps we could come up with a name to make it a bit less verbose than swift package... something like spm that would server as a "type alias" for swift package.

Documentation

Rust: Rust has a rustdoc tool to generate HTML documentation.

Proposal

Something like this would be a big win for Swift. Similar to Rust, this would pull markdown formatting from comments to generate HTML documentation.

Benchmarking

Rust: Rust has command-line benchmarking tools built in.

Proposal

Swift can do this in Xcode, but it might be a good addition to be able to do from command line as well.

Testing

Rust: There really isn't anything specific to call out about Rust testing. This one is just about Swift and XCTest.

Proposal

While XCTest is a perfectly acceptable testing tool, it doesn't feel particularly "Swifty" or expressive... it feels like a bit of a carry-over from Objective-C days. It's fine, just not cohesive with the rest of the Swift experience. I think Swift is due for a more expressive default testing package. There is already a great example out there called Spectre: (GitHub). It's very expressive and makes writing tests feel more like writing Swift. Again, great, but would be even better if it was the default.

Conclusion

So, all in all, Swift is the best language out there. But, there are a lot of developers who don't know that because they aren't iOS/Mac developers. I'd propose we start thinking a bit about the experience of the Swift ecosystemoutside of that iOS/Mac developer community and make Swift's ecosystem as cohesive and friendly as Rust.

55 Likes
2

To be honest i would prefer if the Swift community won't try to emulate the Rust/cargo and NPM way of distributing code.

Endless dependency trees, long download and compile times, versioning issues, package name squatting, multiple packages for nearly the same functionality in various states of stability, potential security issues in your own applications because it is very hard to check who wrote what when you pull in dozens of dependencies and also don't know if those dependencies are maintained at all.

I'd rather use large frameworks like Foundation and Vapor where multiple people who i trust collaborate to create stable and maintained libraries.

13 Likes
3

I agree, especially regarding Toolchain size (installation process in general) and XCTest. I wrote a while back that I really would love to see yarn like testing in the Swift ecosystem - its output is sooo clear and pretty. I didn’t know about Spectre you linked to above, it looks nice, is it built on top of XCTest or independent of it?

4 Likes
4

One cause of the large toolchain size is that most compiler binaries are statically linked. I think this is for efficiency, because dynamically loading clang/LLVM libraries is extremely costly as there is a very large number of relocations.

  • LLDB Debugger: 204M
  • bin/clang: 158M
  • bin/swift: 179M
  • bin/swift-api-digester: 175M

An other cause for the large size is that it contains libraries for every Apple platform:

  • lib/appletvos: 31M
  • lib/appletvsimulator: 63M
  • lib/iphoneos: 121M
  • lib/iphonesimulator: 93M
  • lib/macosx: 47M
  • lib/watchOS: 30M
  • lib/watchsimulator: 63M
7 Likes
5

On a related note, I’ll add to all these useful thoughts that the reason we don’t consider using Swift in any of our core Comp Sci courses at Macalester is the poor state of support on all platforms that aren’t Apple’s.

Half our students have Windows laptops, and maybe 5% have Linux. Incoming students who are new to programming (and perhaps even a bit shaky with using a laptop at all and not a phone!) need to be able to do a foolproof installation from simple directions over a possibly mediocre internet connection and have a working environment on the first day of class. This is just barely even the case on macOS, but on Windows….

(More details on the non-viability of Swift for college teaching in this older post.)

I’m talking about students here, but I imagine the same barriers to adoption exist for many dev teams. This multi-platform story is in my view the main obstacle to @Chris_Lattner3’s dreams of Swift taking over the world.

23 Likes
6

Of all the stuff I'm proposing above, this is the only one that's probably truly inevitable. I think the caution you bring up is not that we can't have a registry, but that one should be able to opt-out if they don't want to participate.

I think this is likely... because SPM already is capable of running free of a central registry, I'd guess it will continue to have this ability post-registry as well. My guess is in the future, we'll either be able to opt into the registry or opt out.

1 Like
7

Not really sure, but I'd guess it's fairly independent. The syntax is very reminiscent of Jest or Mocha/Chai, which are far more expressive (one area where we have a few lessons to learn from JS). Another nice capability these offer is nesting test suites.

8

This makes sense. There is always a push/pull and a reason to do so. I still wonder why Rust is able to be so much smaller but is still quite efficient. Likely because Rust isn't trying too hard to be interoperable... it's just its own thing and only has itself to worry about. Vs. Swift that's interacting with a lot of C and Objective-C.

However, what I wonder is if it would be possible to make a "light" Swift toolchain for devs that weren't interested in iOS/Mac development and the "full" one as well. The light toolchain would be a bit more like the Rust... good for command-line tools, server-side, etc. but could not compile a Mac/iOS app. Mac/iOS app developers would opt in to the full toolchain size (which would probably happen automatically if you install Xcode). 1.6 GB isn't too bad when you consider it can build out fully functional apps for several platforms... excessive when you just want to build and compile a command-line tool.

Having a light version might be a good "gateway drug" into Swift for some non-iOS/Mac developers... and they can always upgrade later if they want to build apps.

9

Which is a shame, because from a language design perspective, Swift is the best thing out there.

Macs are getting pretty universal for a lot of dev teams... but even on Mac, not everyone wants to develop for iOS/Mac or fire up Xcode. Many devs are very attached to VSCode or one of the JetBrains IDEs. While Swift works pretty well on VSCode, it's not well known. Separating Swift a bit from Xcode helps clarify its multi-purpose capability.

But, beyond Apple platforms, Swift has so much potential for:

  • Server side
  • Command-line tools
  • Machine learning (Swift for TensorFlow)
  • Web apps in Swift WASM (I think this is going to be a big one)
  • Etc.

Essentially, Swift is already beloved by the Apple platform development community. Mission accomplished. Now, we need to think about the rest of the world and how to bring them as well!

And yes... embracing Linux and Windows will also be essential for world domination!

FWIW... I think the ability to write Android and Windows apps in Swift would be an asset to Apple platforms. For example, almost all mobile app developers know the need for both an iOS app and Android app. It's just a reality. If could write your app natively for iOS and then easily port an Android version from that, you'd get a lot for developers switching over to Swift/Native iOS... vs. opting for React Native, Flutter, etc. While it may feel to Apple like "supporting the competition", what it's actually doing is enticing developers to develop using Apple tools by reducing some of the friction of going to "Native". But, that just my opinion.

7 Likes
10

I think any sort of registry needs something on top of the basic versioning functionality you get with npm and similar.

I don’t know what it is, but it feels like some sort of developer registration and signing would be appropriate.

All the problems mentioned are real and important ones — security being a huge one, namesquatting, and abandoned code are just a few.

I don’t know what the “fixes” would be, but I think it is better to go slowly and work toward better solutions than to hurry and and throw something together which will become the de-facto standard.

I do like the idea of larger frameworks also — probably just from iOS and Mac background — but there needs to be a way to have better trust in your dependencies. I know big ones that many people work on and many others use can still have issues or could have some malicious dependency of their own, but it seems like fewer to pay attention to maybe?

1 Like
11

Having a light vs heavy fractures the ecosystem. What would you leave out? Then you have code that’s incompatible etc.. probably somebody more knowledgeable than I has something more intelligent to add, but on the surface this seems like a trade off that provides little benefit.

What is the goal? To have a smaller download? Install can always be made fairly easy for the end-user. I do think that would be a great place to focus some effort — to make the non-Apple platforms much easier to get started with.

3 Likes
12

Possible, but not sure it would really have "incompatible" code. If you had the full toolchain (as today), you could build anything. The "light" version would have limitations (likely lacking Mac/iOS developer abilities), but you could always upgrade if you wanted to remove your limitations.

Perhaps it would be more broken into modules vs. multiple versions. So, "light" would be the minimum module. You could add on modules as-needed: Apple Platform support, C interop, TensorFlow, WASM. Today, the toolchain gives you Base + Apple Platforms + C, etc. .. but not TensorFlow (currently a unique toolchain). In this modular approach, you add on piece parts as needed.

Of course, I don't know if this is actually possible or practical.

Yah... smaller download or rather less disk usage. A hesitant developer just dipping their toes in the water may decide 1.6GB is more space than they care to take on their drive. Vs. Rust, where < 200MB is relatively trivial. Ultimately, it's prob not going to be a deal-breaker for most... but could be a hurdle... so the slimmer we get, the lower the hurdle. In the end, you're right... the MOST important thing is an easy installation experience.

I think 1.6GB wouldn't be as noticeable if you downloaded a small toolchain manager ("swiftup" or whatever it's called) and it automated downloading and installing the toolchain for you. It all happens behind the scenes vs. downloading a large installer.

2 Likes
13

And SwiftUI is paving the way. :slight_smile:

Speaking of WebAssembly, which is a core subset of the JavaScript language, the Swift language specification could take a layered approach to lower the threshold for language adoption: The core profile would specify a light and easily portable language, suitable for programming microcontrollers. The higher-level profiles would specify the standard libraries, package management, an extension mechanism, platform-specific extensions etc. When Xcode gets installed, it would add its experimental next-gen language features to the existing Swift environment via the language extension mechanism.

2 Likes
14

Yup... this is exactly what I'm talking about! Allowing a BYO Swift where the developer can get all the components they need and none they don't. Personally, I'll opt for the works, but some won't. Some devs will never want to make an iOS app (or Android app or any app for that matter)... they are interested in low-level, server-side, data science, API development, etc. I think we'd get more of these developers hooked on Swift if they didn't feel they had to load a bunch of Apple platform frameworks on their system to do something unrelated (building machine learning models for scientific research, etc.). A flexible system would best serve all types of Swift developers.

My only addendum to this is this should not simply be via Xcode only. Yes, when Xcode is installed it should do all this for you as well, but by calling underlying command-line tools to manage your Swift toolchain(s). That way it's flexible: developers more comfortable with Xcode could do all this in Xcode, but other developers could do all of this and never even install Xcode... just set up Swift for VSCode, Atom, etc. (Which is also the way it would work on non-Apple platforms anyway).

15

That would imply that Apple uses the same Swift version than anybody, and not a customised internal branch. Which I doubt would be something accepted by Apple.

I think that any discussion about the best way to install and managed Swift should not consider Xcode. If some day, Apple choose to integrate that tool in Xcode, it would be fine, but we should not rely on this design this feature.

16

Ahh... I see. My assumption was that Xcode used the same toolchain as downloaded from Swift.org. That makes sense, but feels wasteful to keep two copies... one for Xcode and one in Library/Developer/Toolchains... However, if modularized as discussed, and Apple has proprietary add-ons for Xcode, these could be more "add-ons" to the base toolchain components. It makes sense that some Mac/iOS stuff might be proprietary, but the base functionality should be the same as open-source.

17

For some more comparison, I looked size for Java.. JDK 14 was ~350MB and JDK 10 was ~500MB. However, where 1.6GB stars looking rather light is at the Android SDK, which tallied up to 16GB (~13GB of that is system images)... but still ~3GB worth of other stuff. Of course, that's apples and oranges... Swift is a language and Android isn't... but the size of Android points to a common source of bloat: platform libraries. This bloat can't really be avoided to some degree. All I'd argue is that it should be more separated from the base language toolchain. Effectively, the Swift toolchain in total will always be large if you want to write iOS apps, just as your Java/Kotlin/Android total size will be large for Android dev. The current assumption (on Mac) is that a Swift developer is a Mac/iOS developer (and largely this is true today). So, if more layered, a base user may not have any Apple platform libraries added until they started their first iOS project (or had any project that relied on these). For iOS/Mac developers, the experience would be exactly the same... but would lighten the base toolchain for developers using Swift purely for research, command line, etc.

Another source of size seems like one that may diminish over time: some of what's needed to interop with Objective-C. I feel like some of this may go away as legacy APIs are updated for Swift and Swift just has less need for bridging legacy code. Probably the biggest thing Rust has going for it in toolchain size is that Rust is a pretty blank-slate language. It wasn't created to bridge anything and has little or no legacy to contend with... vs. Swift that has a lot of Objective-C and C legacy to deal with. Rust just has to deal with Rust. Swift has a much more complicated situation. But again, I think there's hope this may simplify as legacy becomes less relevant.

18

I wouldn't count on Objective-C and C legacy going away any time soon, if ever. Support for that bridging is deeply intertwined in Darwin Swift, as well as the stdlib and Darwin Foundation. To re-architect Swift along the lines that you suggest is going to be an expensive and long-term project that probably has repercussions for clang and LLVM, as well as other LLVM-based languages.

3 Likes
19

There are not two copies of the Xcode default toolchains, only the one toolchain in Xcode.app. /Library/Developer/Toolchains, on my machine, are the tools chains I have built from experimental versions of clang, LLVM, and Swift (TensorFlow). /Library/Developer/CommandLineTools, if you have installed it, has a Unix-y environment that is easier to use from the command line than from within Xcode. But, CommandLineTools is entirely optional, and can be removed with impunity if Xcode is installed

20

Fair enough... Though one glimmer of hope of doing (some) of this could come (if) the Swift complier is ever re-written in Swift... which is also an expensive, long-term project that is unlikely to happen for a while (if ever).

1 Like