Swift 4.2, Package.swift and using Swift as a markup language

Development Package Manager
1

I was trying to use the new Random Unification features of Swift 4.2 and when trying to build it on Linux it was throwing an error telling that the new features were not available. After some debugging I discovered that by default Swift PM compiles with the lowest Swift version set by swiftLanguageVersions or the swift-tools-version on Package.swift.

The underlying problem is stated in the Package Manager Swift Language Version API Update Evolution proposal:

Existing packages will not be impacted but they will not be able to use Swift language version 4.2 unless they update their manifest to tools version 4.2.

This implies that if we want to keep support for Swift 3.X and 4.X and allow the package to build with new Swift 4.2 features we need 3 (!?) package manifests:

Package.swift
Package@swift-4.swift
Package@swift-4.2.swift

This is only one of the great problems derived of having decided to use Swift for defining the Package manifest. And if we keep trying to use it we well end with tons of manifests to maintain. Swift is a programming language, not a markup language. IMHO it was an huge mistake to accept it for defining the package manifest and we should move towards another solution as soon as possible. Of course this is only the tip of the problems it creates, other examples are:

  • Security (reading a simple manifest should not require using an execution sandbox (!?), which I am not sure how/if it is implemented on Linux and other platforms)
  • Reading the manifest requires Swift PM libraries or executable.
  • Mechanically editing the manifest is extremely complex and error prone (hopefully, not only Swift PM would want to read/write it!). Also, requiring SourceKit/SwiftSyntax to edit it feels to me like using a sledgehammer to crack nuts.
  • Creating manifests forward and backward compatible is impossible if we want to add any feature in the PackageDescription API.

For all of these problems I think we should accept to move towards a new simple markup language-based manifest (in .yaml, .json, a new .swiftml/.swiftpkg format or whatever).

I hope this helps reopening this debate as I think propagating the error of using Package.swift is worse than redesigning the Package manifest once more.

6 Likes
2

I believe we're headed in a good direction with a Swift based manifest. The current implementation has several issues but I think we can solve them! This document contains the historical discussion about using a Swift based manifest.

Right, the situation for supporting multiple version of the package manager is not good at the moment. I hope to propose an enhancement in SwiftPM to make this easier because it is generally desirable for libraries to support last few releases of a language. I've not really thought about the possibilities here but I hope we can come up with a good solution!

The sandbox is currently on macOS only. One of the goal of SwiftPM is to provide a tight security model for all operations. This will be especially important once we have extensible build tools which will allow other tools to be executed during a build operation. The same model can be used for reading the manifest securely on all platforms.

I am not so sure if that a major concern. SwiftPM will provide APIs and commands to read or dump the manifest in a well-known format. This will allow other tools to read it.

We definitely want support for mechanical editing of the manifest. This is indeed complex but SwiftPM will handle the complexity for you and it'll expose APIs and commands to manipulate the manifest.

I think this is covered by the "support multiple version of the package manager" point I mentioned above.

1 Like
3

But, how it could be better? I mean, if we continue using a Swift-based manifest it is impossible to design a PackageDescription API which wouldn't become a monstrosity while being backwards and forwards compatible.

Also, all the exposed problems would be extremely easy to solve using a proper markup language instead of Swift. Maybe I am not understanding something, but for me it seems stubborn to maintain Swift as the manifest language after detecting all these problems. What is the great benefit of using Swift-based manifest? That it is a known syntax for the developers?

If we were using JSON or YAML, Swift PM would already have support for adding dependencies mechanically as it would be trivial to implement, there wouldn't be any need for a sandbox, it could be read from IDEs without requiring Swift PM and they would be backwards/forwards compatible by design! :slight_smile:

3 Likes
4

From a formal standpoint, having something turing-complete as your package manifest language seems kind of overkill and prone to bugs.

See: JS vs CSS. CSS is so much better for stable, complex styling precisely because it's not turing complete. You can avoid so many obscure bugs if you forfeit the ability to compute anything inside a stylesheet, which we currently do!

But then again, unlike JS files used wrongly for CSS-tasks, the Swift package manifest files tend to be short. Therefore, their surface area is small and the potential for bugs is lower.

It still screams "overkill" to me. What are the main reasons for having the manifests written in Swift?

And does it give us an advantage from the point of view of expressiveness?

3 Likes
5

I think someone's implemented Rule 110 in CSS.

1 Like
6

Indeed. But is the program capable of running an arbitrarily large board, given sufficient memory?

If so, then yes, CSS is turing-complete. But if you read that program, you'll see that the size is hard-coded. That is not turing-completeness; that is, at best, a finite automata.

1 Like
7

I work in AppCode team, and I think it's worth sharing here an opinion of tool makers (hope, other guys working on Swift integrations for various IDEs and text editors will participate in such discussions as well).

For now, the Swift Package Manager for us is not only a manifest for building the project. Together with the source layout structure, it's a project model. The project model is a core of the resolution, that is required to have a completion, navigation and highlighting in the IDE or any editor.

It means that if you are making an integration into another IDE/text editor, you already have an engine to parse the package manifest and resolve all the entities inside - the SourceKit. And here you don't need to implement or integrate another parser for another language or your custom format. Also, if you don't want to use the SourceKit, you still will need your own Swift parser and resolution engine for parsing the source code, having a completion and navigation. It means, that even in this case reusing it for reading the project model is much better than having another parser just for the project manifest.

Having another language or changing it now will break all the existing integrations using the Swift PM as a project model and I think will have a negative impact on those who maintain such integrations.

Also, having another markup language won't solve such problems as having another version format specifically for the Swift language version in SE-0209. You still need to handle the situation when a new format is not recognized by your parser when processing a value or an attribute for the markup language.

Btw, for us SE-0209 means handling of 2 new code constructs (vs reusing the existing Version struct ). So, from our point of view, the consistency with C++ version is worse here.

8

It still screams "overkill" to me. What are the main reasons for having the manifests written in Swift?

And does it give us an advantage from the point of view of expressiveness?

FWIW, I'm partial to using a markup language like json because of the simplicity of parsing, manipulation, and stability, but Turing completeness would be nice to implement build scripts with (see npm run X).

1 Like
9

I'm fine as using swift as spm language, but a little concerned about the verbosity/complexity it has brought in (and which will increase over time).

For example, here is the simplest Package.swift we can write:

import PackageDescription

let package = Package(
    name: "Stencil",
    products: [
        .library(
            name: "Stencil",
            targets: ["Stencil"]),
    ],
    dependencies: [
      .package(url: "https://github.com/kylef/PathKit.git", from: "0.9.0")
    ],
    targets: [
        .target(
            name: "Stencil",
            dependencies: [])
    ]
)

Which does not make simple cases (declaring a library) simple nor does it handle well complex ones (multiple packages repository).
Not sure we would have gone that way with a much simpler JSON format.

10

JSON has many other issues, no support for comments being one of them. And while there arguably are markup languages fit for the job of manifest files (see Rust using toml for instance), the idea here is to think of the Package.swift as a two-part manifest in the future.
A very declarative first part that maybe doesn't allow dynamic contents and is easier to edit (more so when spm gains support for handling all that so other tools don't have to be built - I've tried my hand that, wasn't so fun :smile:) and an optional second part for doing whatever a package author has to do to support their package. Most packages wouldn't need that, again see Rust's build.rs as an analogy.

11

Using Swift to describe a package has the benefit that anyone should be able to understand it, and swiftc can check the format.
Still, I've been quite surprised that Package.swift is actually interpreted by a full fledged swiftc, instead of a simple parser which understands the syntax, but can't wipe volumes or connect to whatever server that got an IP in the internet.

I'd prefer a solution where you have to explicitly request that a package needs scripting (import ScriptedPackageDescription?), and where a user of the package has to allow the scripting.

1 Like
12

It's hard for me to see a real security issue here, since the primary use of a package is in contexts where you're going to compile and run the packaged code anyway, so there's a fairly high degree of trust already involved. I guess an exception would be the edge case of someone building a central package indexing service, where they want to extract information from the package but are not interested in using it.

13

For me, that is more of a fundamental attitude than fear of a specific risk:
I don't think it is a good idea to make execution of code from arbitrary sources a common thing. I'd rather prefer users to stay suspicious when a package requests special rights, instead of getting accustomed to it.

Besides that:
As soon as SPM is integrated into the standard build pipeline for iOS apps, it will be a very attractive target for criminals.
I remember the stories about hacked Xcode versions that would infect apps, and execution of Package.swift looks like a great vector to attack CI servers.

3 Likes
14

This is essentially the entire point of a package manager, though, for better or worse.

Sure, but if you want to infect apps and you have enough access to the package's code to edit Package.swift then just put your exploit code directly in the code of the compromised package and avoid the intermediate step and the sandbox.

1 Like
15

The difference is that now I can also hack you, not just the built object.

1 Like
16

One runs on the developer's machine during compilation, the other on a possible user's machine during execution of the final product. That's two entirely different targets.

edit: oops, @nick.keets was faster than me :see_no_evil:

1 Like
17

Sure, if you're developing an iOS app and you trust the simulator sandbox more than the SPM sandbox. I didn't say there was no extra attack surface, or that they were somehow identical, just that the difference is very narrow and infecting the iOS app is probably more valuable than a single developer's machine, etc.

18

That's a common approach to security nowadays. To paraphrase: since we can't be 100% secure, let's don't bother making anything secure.

Having an SPM exploit would enable exploiting people just by testing a cool new lib they found on Github. No need to even ship an app that will take time, beta testing, review, etc and will present a lot of opportunities to discover it. Taking advantage of SPM would give instant gratification.

2 Likes
19

I think this topic got a bit derailed from the original point regarding Swift as a language for the manifest. A discussion on package manager security is definitely interesting as well, but I don't believe these two necessarily have a lot to do with each other. For one, manifests are run in a sandbox (on macOS at least) preventing the most obvious of exploits and on the other hand other package managers using non-turing-complete languages for their manifests have a similar problem, npm's install script hooks for example (or a super small proof of concept for Carthage :sweat_smile:) .

Once SPM's manifest format gains a few capabilities for limiting what's possible (possibly per section as I stated above, which has been mentioned before in other threads several times), SPM can start to limit what a package's dependencies are allowed to do or implement a completely different kind of trust model, whatever that may be based on. I'm pretty sure this is not the first thread on that topic either :wink:

2 Likes
20

A build server does not even have to run its product - yet it is a more attractive target than a single developer machine or a single app:
It allows you to infect all apps that are packaged by the server, and you can get malicious code into the main app of the owner of the CI via a small toy project that happens to be using the same infrastructure.

I certainly would not want my CI to silently execute code that's coming from a dependency of a dependency...

3 Likes