SE-0272 Package Manager Binary Dependencies

Evolution Proposal Reviews
21

This proposal is specifically for Apple platforms only, though, so you won't have any binaries when doing development on Linux anyway.

1 Like
22

Regarding, the discussion around allowsBinaryDependencies: Bool. I think you are right we should change the name of this configuration property as long as there is no problem with having the default to true (which there shouldn't be).

My thoughts on the opt-in: I would rather not include this in the initial proposal since this can be a purely additional feature that can be proposed afterwards. Once we have the integration for binary dependencies we can propose to add a whitelist of domains to a configuration file which SwiftPM will take into consideration or as @NeoNacho said we can leave this part up to the workflow tooling.

23

Flag to allow binary dependencies

I agree with the general sentiment so far around renaming disallowsBinaryDependencies to reflect the positive statement, e.g. call it allowsBinaryDependencies.

However, to take a step back, I still think this belongs in the config file because I don't think the author of a package is the best authority to determine whether the package's dependency graph prohibits binary packages. The author of the root package is really in the best position to make this policy decision.

For example, consider that A depends on B, and A disables binary packages. If I fork (mirror) B to add a dependency on a binary package, I now need to fork A as well solely to remove the prohibitory flag, which really should not be necessary. I know this is not really the intended use case of mirroring, but it still gets used in this manner, and represents a potential nuisance for developers.

I'm struggling to find a real world use case for a package author to impose a restriction on binary dependencies to their clients, within the dependency sub-graph of that package and its dependencies. The package author can't possibly anticipate the needs and policies of their own downstream clients in this way. If anyone can think of counter-examples, please call those out!

New PackageDescription API

The checksum parameter of the url case in Artifact.Source should itself be an enumeration rather than a raw string, for example:

enum Checksum {
    case crc32(String)
}
case url(String, checksum: Checksum)

This would provide:

  • Semantic meaning to that checksum string, i.e. what kind of checksum it actually is
  • Future expandability if we want to provide alternative checksums formats in the future, such as cryptographic signatures
  • Ability to react to changing security realities, if the checksum algorithm is found to be broken and producing checksum collisions becomes possible, we can deprecate the enum case for that algorithm and introduce alternatives

I also did not see that the checksum format was defined in the proposal, which such an API would do so implicitly.

Artifact.Source enum's path case

This has no associated String parameter for the actual path of the binary artifact (the path to the .xcframework in this case). Was this an omission or is this expected to be implied somehow?

Detail of the proposed APIs

Can you please provide more complete Swift interfaces of the manifest API changes in the proposal? For example, it's not clear what the signature of the .binaryTarget initializer is, or why artifacts is an array (since the .artifact initializer doesn't demonstrate any sort of condition mechanism).

I'd also like to see the artifact format fully described -- the proposal notes that XCFrameworks are used, but doesn't state the format of the artifact itself (it's implied to be a .zip?) or how the internal contents of that archive are expected to be laid out.

Other

I think this is out of scope for this proposal. As @NeoNacho states, binary packages can also be pinned to specific versions, and changes to the Package.resolved should be a sufficient trigger to perform manual judgement on any updates to the package graph.

I'd also caution against filing this under "security" concerns since we generally concluded in the original pitch thread that binary dependencies shouldn't be considered a security issue; it's more of a policy issue. The problem with codifying these kinds of restrictions is that different users will have different needs, so it makes more sense to push this problem out to external tools. For example, users may:

  • Want to prohibit all binary packages
  • Want to allow binary packages only from a certain vendor
  • Want to allow binary packages only downloaded from certain domains
  • Want to allow binary packages only under certain licensing terms
  • Want to allow binary packages only if the artifacts are provided via local paths in the package
  • Some combination of the above

Because the complexity of these sort of business requirements is potentially unbounded, I don't think they belong in the base tool.

Overall, great progress! I'm very happy to see this nearing closer to completion as it's a significant step towards feature parity with other package managers in the ecosystem.

7 Likes
24

Overall, this proposal seems to be going in the right direction. I haven't closely followed the discussion so I might be repeating some of the points already made but here's my feedback:

Option for disallowing binary dependencies

I don't think disallowsBinaryDependencies really adds any practical value and it seems overly restrictive if a package author wants to override a dependency that is present in the subgraph of a package that uses that flag. I suggest just dropping that functionality and we can consider adding workflow features in the future if that's something users end up needing.

Checksum

SwiftPM needs to provide some way of computing the checksum since it's needed in the package manifest. I suggest proposing some command like swift package compute-checksum <file> so the actual checksum implementation can be evolved in the future. Also, the checksum implementation should be tied to the tools version of the package to avoid breaking compatibility with older tools.

I don't think we need to store the checksum in Package.resolved file since it's already present in the package manifest file. We already store the git hash in the Package.resolved file and we can start validating that hash for packages with binary targets. If the validation fails because, let's say someone
publishes a tag again for genuine reasons, SwiftPM can surface that and provide a way to override the validation temporarily.

Manifest API

Given that this proposal only plans to support XCFramework right now, we can simplify the manifest API to:

extension Target {
    /// Declare a binary target with the given url.
    public static func binaryTarget(
        name: String,
        url: String,
        checksum: String
    ) -> Target

    /// Declare a binary target with the given path on disk.
    public static func binaryTarget(
        name: String,
        path: String
    ) -> Target
}

Then the example usage becomes:

let package = Package(
    name: "SomePackage",
    targets: [
        .binaryTarget(
            name: "SomePackageLib",
            url: "https://github.com/some/package/releases/download/1.0.0/SomePackage-1.0.0.zip",
            checksum: "<something>"
        ),
        .binaryTarget(
            name: "SomeLibOnDisk",
            path: "artifacts/SomeLibOnDisk.zip"
        ),
     ]
)

Some other thoughts

  • We should probably support pointing directly to XCFrameworks for path-based binary target API since that'll be helpful for package authors during development.

  • The proposal should clarify the expected contents of the zip file.

  • Perhaps, SwiftPM should also emit error or warnings if there .swiftmodule files in the XCFramework since that is not a stable format.

  • SwiftPM should emit an error if a package directly exports a binary product and specifies a type. For example, it should be an error to write .product(name: "MyBinaryLib", type: .static, targets: ["MyBinaryLib"]).

  • swift build --target <target-name> should reject binary targets.

  • We need to figure out the behavior when a regular target that depends on a binary target is built for non-Apple platform. Maybe SwiftPM can just ignore those dependencies since binary targets will be currently only supported for Apple platforms?

6 Likes
25

  • Option for disallowing binary dependencies: One such use case is mentioned here. With the intent being that the package author doesn't accidentally add binary dependencies, we could say this flag is ignored if the dependency is used transitively to resolve the overriding issue. I don't feel strongly about this flag, though. If the general consensus is that it doesn't provide enough value, we should drop it.

  • Checksum: relying on the tag seems to add much more complexity, especially if we want to provide an additional override. Are there any concerns with storing the checksum?

  • Manifest API: I like the suggested simplification, the current API has evolved from a more complex one and it seems good to clean that up since we removed that complexity from the proposal.

  • I don't think we should get into the business of verifying the contents of the XCFramework. If we think that's important, there are more conditions we should verify as well, such as whether the module name of the binary matches what is specified in the framework.

  • The type is actually an interesting question we hadn't considered, yet. Presumably .automatic should actually be an error and we should require the type to match whatever linkage the binary artifact uses. SwiftPM could then verify it, if we think that's important.

  • I don't think ignoring dependencies makes sense and that seems unexpected. Ideally, we would have solved that through platform restrictions already. My expectation is that there's no special behaviour for non-Apple platforms, but instead we will enumerate what the artifacts provide and fail the build if there's no binary for the destination. This would cover both non-Apple platforms (where this would always fail today), as well as the case where the XCFramework doesn't contain a binary for the destination.

26

That would then make it a source‐breaking change to add support for other platforms, since what was once safely ignored would suddenly be expected to succeed.

Maybe it would be better to first add a way to explicitly hide a target from a particular platform and to depend on another target only on certain platforms? (Such a feature would be useful for so many other things too...)

It would be annoying that the inclusion of such and artifact would break the entire build graph for the other platforms, even if they aren’t trying to use anything provided by it.

At the same time, the platform conditions necessary to make it work do feel like they belong as a separate feature with a separate proposal. Maybe with some assurance that it is planned for some point in the future, we can just accept that for now binary targets will suffer from some extra usability constraints until the other API materializes to solve them. Then we can move forward now with the binary targets without worrying about it.

27

My gut feeling is that this isn't an immediate problem. If we look at existing binary-only frameworks in the ecosystem, they are mostly used as top-level dependencies or by libraries which use them as core functionality. I think it is not very common for a library to use a binary-only framework as just a minor optional feature for some platforms. If this assumption ends up being wrong, we could think about providing ways to conditionalize dependencies as you mentioned.

28

I wasn’t so much thinking of minor optional features so much as the ability to write something on top of different binary dependencies that are written for different platforms, but offer similar functionality. You can easily use #if to switch between AppKit and UIKit in code. But if they were packages instead of system libraries, you still couldn’t build for either macOS or iOS, because the package graph would always require the impossible build of one of the dependencies for the wrong platform.

(With source targets this problem can be worked around by surrounding each and every file with #if so that it can still be built, but that isn’t an option for binaries.)

1 Like
29

disallowsBinaryDependencies flag

I also don't have a strong opinion on keeping this flag. In the beginning this was meant to give package authors an escape hatch; however, as @jakepetroules mentions this can cause weird behaviours for transitive dependencies and in mirroring. I agree with @NeoNacho that we can drop it if we don't see a big enough need and solve this through workflow tooling as well.

Checksum

I think adding the checksum generation as a separate swift package command and tying it to the tools verison makes sense. However, regarding @Aciid point about not storing the checksum in the Package.resolved. You said that we have the git hash already in there but for binary dependencies we don't have such a hash since we are pointing to an arbitrary URL. I still see value in having the hash in the Package.resolved additionally to the manifest file.

Artifact format

Since we only support Apple platforms from the beginning, the proposed format would be a .zip file that contains a single XCFramework. The vendor needs to make sure that this framework is valid and supports the needed platforms. I would not want to go into validating anything of this framework just that there is one in the zip.

Manifest API

@Aciid You are right after dropping the complex Linux support we can simplify the APIs. The only thing I want to make sure of is that we are still able to evolve it to a more sophisticated version in the future. With dropping the whole Artifact and Source struct are we still able to do this? Would we just introduce a new tools version and deprecate the old ones as soon as we introduce Linux support?

Non-Apple platforms

I think as the first step we should not think too much about packages that have binary dependencies and want to support non-Apple platforms. I agree with @NeoNacho that in the beginning these would mostly only be used as top-level dependencies. For me it is important that we solve the case of Firebase, Adjust and all the other vendored frameworks on Apple platforms. But still keep the possibility to extend this API to conditionalize the binary dependencies for different platforms. In my opinion, this is the most pressing problem we have to solve at the beginning to replace Cocoapods and Carthage with SwiftPM.

2 Likes
30

I don't have a particularly high stake in this matter, and I can understand the pragmatic reasons for wanting to restrict this to an Apple-exclusive feature at first, but I worry a bit that introducing this will create another rift between Apple and non-Apple platforms. I could foresee people creating useful packages that have binary dependencies and will then only be available on Apple, which I would find very unfortunate.

3 Likes
31

The proposal doesn't mention anything about mirroring URLs (like https://github.com/apple/swift-evolution/blob/master/proposals/0219-package-manager-dependency-mirroring.md).

Should we be able to mirror the URL of a binary artifact, like we do with a git repo? I guess it is not entirely necessary, since you could still mirror the whole Package.swift and replace the the URL in the mirrored repo, but it is something to discuss.

32

It will depend on the concrete design, but we can definitely change the API based on the tools version and deprecate the current one if needed.

33

This is a good point. I think we probably should allow mirroring artifacts as well since mirrored repos shouldn't need manifest changes to work. Modifying the manifest would also likely be a manual process.

34

The review of SE-0272: Package Manager Binary Dependencies ran from November 13th through November 20th, 2019. The proposal is returned for revision. Thanks to everyone who participated!

Boris Buegling
Review Manager

35

Hello,

I think I am a little bit late to the discussion.

I think that the binary artifact proposal is too tied to the specifics of the Apple platform and this had bled into the Linux supports - where the proposal does not attempt to support Linux given that there is no ABI stability.

I believe that it is important to support binary packages as part of the build process, even if those binaries are not generally useful to the world at large. There are indeed many Linux distributions, but developers typically choose a couple of popular distributions/versions for their binary payloads, or they could provide their binary payloads on their own.

For example, PyTorch or TensorFlow are both shipped for a handful of Linux distributions - certainly not every distribution out in the wild, but a good subset. I would love to see the package system allow me to at least let me consume those binary payloads at my own risk (that is - I ensure that the binary matches the host I am running on).

This last point is key and leads to another capability that might be useful - it is not just a matter of consuming a publicly published binary artifact, but the possibility of referencing and consuming binary artifacts that I might have produced on my system through other means (for example, I might have used my own experimental build system that happens to output a set of header files and native libraries).

I would not like this new binary artifact capability to be kept away from those of us Linux users that just want a way of consuming these libraries for Swift on Linux - in fact, I would say that this is probably more important than the Apple platform case, where there is already a system to create Swift libraries that consume binary artifacts of unknown provenance.

36

The problem with this is that you need a way to communicate that you're accepting the risk: that you're writing a SwiftPM package that says "I promise that if my code fails to link I won't blame my build system but only my own self." I am very reluctant to have SwiftPM packages that only sometimes work: in my mind it leads to nasty outcomes and increased support costs.

For local SwiftPM packages I think your wider build system can be responsible for pulling binary packages for Linux. For packages you want to distribute as a library it is very hard to communicate your constraints with this proposal, and to communicate those constraints properly requires quite a lot of work.

I also want to see a solution for Linux, but the authors of this proposal were fairly clear that they were not willing to take on the full requirement of specifying what a Linux platform looks like in order to do this properly.

37

I think the use case for libraries that you built yourself and installed on the system should be covered by system library targets already.

I agree that we should eventually solve this problem for Linux and other non-Apple platforms as well, but scoping down the proposal to something we can deliver in the short term made sense to me.

2 Likes
38

In our internal build tool, we pass a sort of platform identifier to the build command so it knows which one to build. Of course we have sensible defaults (e.g. "apple" for iOS and Mac, "android", etc.) on supported platforms. However, you can override this and be explicit.

Maybe that would be a workaround that would allow supporting multiple platforms in an "ad-hoc way" and say that you accept the risk? I.e. a --platform "org.ubuntu.breezy.x64" switch that can be passed to SwiftPM, and an intentionally very simple annotation for a binary dependency which platform it is?

Then we could define official platforms like com.apple.ios.arm64 (a comma-separated list of such platforms could be used to mean "I have a universal binary for these platforms"), and for unsupported platforms, people would be advised to use their own identifiers. If all the specified platforms match, a binary dependency is used, otherwise skipped.

We could always in the future add more smarts via special keys (like .hasfpu or whatever makes sense), but we would have a backwards-compatible, extensible way to resolve dependencies until then.

39

I've been looking at the in-progress implementation of this feature, and I have a question: since the current implementation requires the use of XCFramework, how is it going to handle resource bundles?

The current proposal for resources support appears to be generating a static library alongside a bundle for the resources. But I don't see any support in XCFramework for a resources bundle. So, if you have a Swift Package with resources, and you want to distribute it as a binary, how do you create an XCFramework with both the static library and the resources bundle? Is this something that is being considered currently?

40

Shouldn't binaryTargets have a place to add dependencies? Binary targets are already built, but consuming targets might still need the transitive dependencies.

For example, 2 binary targets that point to static libraries:

    .binaryTarget(
        name: "SomeStaticLibraryOnDisk",
        path: "artifacts/SomeStaticLibraryOnDisk.zip", // Static Library
        dependencies: ["SomeOtherStaticLibraryOnDisk"] // The dependency is not linked into the binary, so consumers need to link it.
    ),
    .binaryTarget(
        name: "SomeOtherStaticLibraryOnDisk",
        path: "artifacts/SomeOtherStaticLibraryOnDisk.zip"
    ),
    .target(
        name: "SomeRegularTarget",
        dependencies: ["SomeStaticLibraryOnDisk"] // SomeRegularTarget needs to link SomeStaticLibraryOnDisk, but it should also link SomeOtherStaticLibraryOnDisk since it is required by SomeStaticLibraryOnDisk.
    ),

SomeRegularTarget only needs SomeStaticLibraryOnDisk, but SomeStaticLibraryOnDisk needs SomeOtherStaticLibraryOnDisk. Since SomeStaticLibraryOnDisk is a static library, it doesn't ship its binary with SomeOtherStaticLibraryOnDisk linked in - it just forces consumers to link against SomeOtherStaticLibraryOnDisk as well.

Rather than require consumers to be aware of this, the dependency graph could be utilized to direct SPM to do the right thing automatically.