SE-0450: Package traits

That's currently incorrect, one can't explicitly set everything they want. To configure a package, the only way to provide inputs to a package manifest at build time is Foundation.ProcessInfo.environment (or lower level platform-specific APIs like getenv, which one shouldn't use in package manifests anyway).

This has multiple downsides: it isn't deterministic (thus breaking incremental builds), it is stringly-typed, there's no way for a package to declare its configuration points upfront. That makes environment variables significantly more implicit than the proposed traits API.

The traits API is designed to replace that environment variables hack and streamline the whole "configure at build time" approach, while also allowing packages to propagate configuration down to their dependencies.

2 Likes

You are already practically providing a diff-based API. I'm mostly saying that it should have both legs, not one, and also that it can perhaps be expressed in more Swifty APIs with the help of an [structified-]enum.

The proposal APIs allow default-traits plus other-traits, but they don't allow default-traits minus other-traits. I'm arguing that I foresee both legs will be beneficial, not just the first one.

It can be a big trouble to provide "default-traits minus other-traits" APIs later on when the APIs are declared stable, so we need to come to a conclusion rather sooner than later.

Thanks for confirming!

One more question around mutually exclusive traits:

How does this square with the requirement for documentation generation that all traits must be enabled so that the generator can see every API that might be available? Since doc generation requirements typechecking and compiling the module to extract symbol graphs, I presume that the code above would also cause a build error, preventing docs from being generated. Would a package in such a configuration simply be unable to generate documentation?

Maybe that's a decent way to enforce that well-behaved packages in the ecosystem—who presumably want to generate docs for their users—don't write code that violates the assumptions of traits, though...

That is correct, the API is designed to be explicit with the principle of least surprise in mind. We don't want package consumers to be confused when they were unexpectedly opted into some trait without their knowledge, if some transitive dependency added a new trait or two.

Using this analogy presupposes that both APIs are equivalent or symmetrical in some way, but explicit != implicit, and only one can be built on top of the other.

It remains to be seen if enough package consumers find a need for some additional implicit API. We have to land a more primitive explicit API first and spend some time gathering feedback, before building higher level abstractions and baking in implicit behaviors too early.

1 Like

I'm struggling to understand how that is different for the 2 cases.
So you're saying "default-traits plus other-traits" is good and explicit but "default-traits minus other-traits" is bad and implicit?!
As far as I can tell both will implicitly opt you into the default traits that are added later in the package's lifecycle.

It should be a package author's responsibility to decide whether they think their users will be "surprised" by a new enabled-by-default trait or not.
SwiftPM might want to help and make sure users can completely opt out of default traits, but I don't think that should be the first thing to do, as I think the package authors should remain in command of what their users experience, considering they'll likely know best.

Correct.

That's why the proposal clarifies that default traits are special and involve semver breaking changes.

It's also package consumer's responsibility to ensure the code they're developing doesn't unexpectedly break when they update their dependencies. There should be a mechanical way for a package consumer to acknowledge they want a newly added dependency trait to be enabled or not, if they were using an explicit API in the first place.

At the same time, we need to ensure that breaking changes are clearly communicated to users and users remain in command of what traits they consume. If a package author doesn't know best, a user should have means to override package's bad defaults. That's why providing an explicit API as a building block is crucial.

I'm not against what you're proposing in principle, but I would like to see a significant amount of real world packages showing real world use cases for diff-based API to exist in the first place.

To be honest my main problem isn't existence or non-existence of a diff-based API. The current APIs already provide "default-traits plus other-traits" and i was just thinking about how we could introduce APIs to get to "default-traits minus other-traits".

I think one of your concerns is that the APIs i suggested don't provide a way to simply just enable an specific set of features that you want, without caring about the default features. So again, those APIs were just what i came up with to get to "default-traits minus other-traits", and I'm open to any other proper API shape as long as we can get that included.

That should be the author's responsibility though. If there is a new trait that would break users, then authors of packages which follow SemVer shouldn't enable that trait by default without a SemVer major bump.

How is default minus an explicitly provided set of traits you want to disable / opt-out of implicit? (Given default + set of traits is explicit)

Might be minor but it is bugging me out not to understand how an explicit list is now implicit :D. What is implicit about it :)?

Thanks for the healthy discussion @MahdiBM. As I said before I do understand where you are coming from and from a package author perspective I completely get what you want to achieve. However, I still believe the proposal here is choosing the right approach.

You brought up one thing earlier that I think is pretty important in this discussion:

I want to dig into this a bit. You are right that you cannot express that but I would argue it's not important to be able to express it. A few reason for this. First, as a package author you should very carefully consider what you make a default trait. It should really be the case that the vast majority of your adopters need all of them. Secondly, as an adopt if you want a different set of traits than the default you probably care a lot about all the traits that are enabled on your dependency including future new default traits. I expect that adopter packages mostly fall into three categories:

  1. Just using the default traits
  2. Using the default traits + some extra ones
  3. Completely customising the set of traits

Now I understand your point that we are not supporting default traits - some extra ones but I really believe that use-case is minimal. Nobody is missing out on not having a default trait enabled since traits are purely additive. If a behaviour of a library should truly be default then it shouldn't be put behind a trait.

Lastly, the most important point for me why disabling traits is a niche use-case is that in the end traits get unified across the whole dependency graph. So while a package might disable a trait, it is no guarantee that the trait is in fact disabled. Another package might enable it anyways. That makes this whole concept of disabling traits harder to explain "Well yes you disabled it but the package over there in your graph enabled it so you have it enabled anyways".

Now, I am not saying we cannot add it in the future. We could totally extend the API with a set of disabledTraits if we wanted to.

4 Likes

It's a good point. Mutually exclusive traits are really a niche use-case. I fully acknowledge that there are some good use-cases but the majority of the packages shouldn't do it. Now for documentation you are right it requires special care. We might need to do a multi-pass documentation generation and merge it together.
In general, I think documentation can evolve further with traits. Looking at the cargo ecosystem initially it wasn't visible that which APIs are gated by which trait. Over time, the Rust documentation tooling got better and now you can see in the docs the traits gating APIs.

1 Like

I disagree there. If there is a problem or something that can be solved by disabling one trait, that doesn't mean I want to carefully choose all traits I want to disable and I care to periodically check what new default features are introduced so I can manually enable them. The current API will encourage users to disable all default features just to do that, which will backfire in form of users missing on new enabled-by-default features.
This is likely to happen in real world. It merely takes 1 enabled-by-default feature that at some point in time starts to cause or enable a problem, which then users would want to get away from, even temporarily.

Nobody is missing out on not having a default trait enabled since traits are purely additive. If a behaviour of a library should truly be default then it shouldn't be put behind a trait.

I'd argue that users will be missing out on those features.
If there is a feature that is nice to have for 70% of the users, but 30% will suffer, then it makes sense to having it as a trait that is enabled by default while giving users the option to opt out of the feature.

So while a package might disable a trait, it is no guarantee that the trait is in fact disabled. Another package might enable it anyways.

That's a good note, but in that case, shouldn't SwiftPM just notify the user instead of "implicitly" enabling that trait? I'd imagine this will be another source of (infrequent) headaches. Something like Package dep-package requires trait 'foo' to be enabled on dependency another-dep-package.

That's suboptimal though. Having both traits and disabledTraits?
Shouldn't we account for such a situation in a better manner?
Like not having traits: Set<Package.Dependency.Trait> but traits: Package.Dependency.Traits which is expressible by array literal?
I guess we always have the option to deprecate old APIs, but that's also suboptimal.

I still think this will be a pretty common situation and prefer to see it addressed with the current APIs as I know swift-package-manager is a pretty big and vital project with comparatively little work-force trying their best on it, so another concern of mine is that we'll be left with the current APIs and I'd think this will just fall out of priority and add another paper-cut to the list of paper-cuts originating from SwiftPM (more on these later, have been planning to do a post).

1 Like

IMO it's a more general problem out of scope of the traits proposal. We already have problems with symbol graph not generated for APIs specific to non-Darwin platforms. If/when that's addressed, I'd expect support for trait-enabled APIs documentation also to be fixed as a special case of that problem.

4 Likes

My 2c is that I fully agree on this and I really like the solution proposed. Any tool I use becomes a pain to maintain and update across versions when options change behind the scenes without having an easy way to control it. Is not exactly the same, but formatting and linting are good examples of how most tools get it wrong imo. I'm glad the proposal allows me to disable everything and control exactly what's enabled, while letting people that just wants a quick start to use the default. :+1:
I understand going the extra step and making enable/disable fully strict would be too much for most people so I won't propose that :P

3 Likes

Something I'm not fully understanding yet (I will have to read the proposal again) and surprised me is the fact that the flags are in the dependencies declaration instead of in the target.dependencies. Does this mean the enabled flags must always be the same for all the targets in the same package?

If by "flags" you mean traits, then yes. In fact they're the same across the whole package graph. Having the same package in the same package graph with different traits would be equivalent to having different versions of the same package in the same graph, which is currently not possible.

You can think of traits as yet another aspect of versioning, they're unified to a single traits-version across the whole graph.

If by "flags" you mean #if conditions, then no, they don't have to be the same. You can declare your own per-module trait-conditional Swift setting:

swiftSettings: [
  .define("Foo", condition: .when(trait: ["Bar"]))
]
2 Likes

I had a quick look over that proposal and have already some use-cases in mind. So it’s a +1 for me.
Looking at the proposal it certainly is quite a big addition and requires careful planning on what to put behind a trait and what not and get it all work correctly in all combinations of enabled/disabled traits, but I think this is a necessary burden for the package developer in making their Package more modular.

In general, I really enjoy the direction SPM is going into for supporting large-scale applications, so thank you very much for implementing this!

Rather than [.default, "SomeTrait"] and ["SomeTrait"], would it make sense to reverse the polarity and have ["SomeTrait"] and [.noDefaults, "SomeTrait"]? Since opting out of the default traits should be less common than enabling a non-default trait with the defaults.

1 Like

I can provide a case study where the package traits really improve both the ergonomics and safety of the package management.

I have a C library dependency that is currently modelled as a system library dependency. Like many standard C libraries out there, this library can be compiled with various optional features, often depending on the presence of other libraries. When I depend on this library I require a specific piece of optional functionality to be present through my dependency. But, there's no way to model this through a system dependency. And so, it's entirely vulnerable to the environment in which the library was compiled with no way to discover the mismatch until runtime, hopefully through some test. This is far too late in the process to find this problem.

It's possible to vendor this dependency as a local target within my package. I can set it up so that the C defines are set to force the optional functionality, and provide the system library dependency to match that. But then there are some problems:

  • Everyone has their own private copy of the library owning the maintenance burden of updating it themselves
  • Library code is copied into each repo using something like git submodule, git subree, or worse, a direct copy that's not as trivial to sync with the upstream library code

If I want to make a version of this library that is shared with the rest of the Swift community I need to take into consideration that the package will need to support different needs in terms of optional functionality. The package can have a different shape for different dependencies. So, how do I provide switches to change its shape today?

The only viable option for controlling a package's shape is through environment variables at the moment. Here's a snippet of a Package.swift that allows its shape to be changed by environment variables:

var libTarget: Target = .target(
    name: "libfoo",
    dependencies: [],
    cSettings: []
)

let package = Package(name: "LibFoo", ...)

// This is a list of system libraries that can be enabled in the libfoo to add extra functionality using the specified environment variable in swift build
let libs = [
    (envVar: "LIBFOO_ENABLE_Z", define: "HAVE_LIBZ", libName: "zlib", moduleLoc: "swiftpm/zlib", pkgConfig: "zlib", aptProvider: "zlib1g-dev"),
...
]

for lib in libs {
    // Until we have traits, we use environment variables here to set which system libraries we want to use
    if let _ = ProcessInfo.processInfo.environment[lib.envVar] {
        libTarget.dependencies.append(.target(name: lib.libName))
        libTarget.cSettings!.append(CSetting.define(lib.define, to: "1"))
        package.targets.append(
            .systemLibrary(
                name: lib.libName,
                path: lib.moduleLoc,
                pkgConfig: lib.pkgConfig,
                providers: [.apt([lib.aptProvider])]
            )
        )
    }
}

While this gets us closer to a sharable SwiftPM for this library there's still a few major problems:

  • The user running swift build must know the environment variables to control the shapes of all of the dependencies
  • The user must know the transitive requirements of all of the packages for their dependencies
  • SwiftPM doesn't know anything about these environment variables, the build graph could become mismatched between invocations depending on the state of the environment variables

So, how can this look if we use package traits to control the package's shape?

// This is a list of traits that can be enabled in libfoo to add extra functionality using the specified trait coming from either a system library or an SDK built-in.
// Dependent packages can enable the traits that they need and/or the `swift build` can enable one or more of them.
let traits  = [
    (name: "Z", define: "HAVE_LIBZ", libName: "zlib", moduleLoc: "swiftpm/zlib", pkgConfig: "zlib", aptProvider: "zlib1g-dev"),
...
]

let pkgTraits = Set<Trait>(traits.map { .trait(name: $0.name) })

let sysLibs: [PackageDescription.Target] = traits.map { trait in
    .systemLibrary(
        name: trait.libName,
        path: trait.moduleLoc,
        pkgConfig: trait.pkgConfig,
        providers: [.apt([trait.aptProvider])]
    )
}

let sysLibDeps: [PackageDescription.Target.Dependency] = traits.map { trait in
    .target(
        name: trait.libName,
        condition: .when(platforms: trait.sysLibPlatforms, traits: [trait.name])
    )
}

let cSettings: [PackageDescription.CSetting] = traits.map { trait in
    .define(trait.define, to: "1", .when(traits: [trait.name]))
}

let package = Package(
    name: "LibFoo",
    traits: pkgTraits,
    targets: [
        .target(
            name: "libfoo",
            dependencies: sysLibDeps,
            cSettings: cSettings,
        ),

Now anyone can depend on libfoo like this, requiring a particular trait:

Package(
    name: "mytool",
    dependencies: .package(url: "https://github.com/bar/foo.git", from: "1.0.0", traits: ["Z"]) // We need the zlib trait because there are zlib compressed data streams that foo will be handling
)

And other dependents can require their own traits of the libfoo also. SwiftPM will be able to have enough knowledge to assemble an appropriate build graph at all times.

The ergonomics of being able to add declarative .when() conditions with the traits on the C settings and dependencies can also make the trait example even simpler, because they could have been statically coded into the targets instead of mapped out with the let statements and mapping. In the environment example above this would not have been possible because of the lack of the ability to tie a when condition to an environment variable.

Finally, I can share this C library with others in the Swift ecosystem!

Overall, I see the traits as a big benefit to the Swift ecosystem not only because it can be adapted to C libraries like in this case study, but also because it allows Swift packages to adjust their shape too based on a variety of factors, such as code footprint (using defines), and controlling optional dependencies.

4 Likes

Thanks, that makes sense.

In a related note, does this help in anyway in avoiding downloading dependencies that are not used because a trait is disabled, or does this only affect compilation? Like for the scenario described

Some packages want to make it configurable what underlying technology is used. The Swift OpenAPIGenerator for example is capable of running on top of URLSession , AsyncHTTPClient , Hummingbird or Vapor . To avoid bringing all of those potential dependencies into every adopters binary, the project has created individual repositories for each transport. This achieves the goal of making the dependencies optional; however, it requires users to discover those adjacent repositories and add additional dependencies to their project.

Is not clear to me that if the openapigenerator (as an example) decided to unify under 1 repo and use this new flag system to differentiate implementations, would a consumer of the openapigenerator dependency that only enables a theoretical URLSession trait, pay the cost of downloading all the other unused dependencies?

IIRC in the current implementation it doesn't, but in theory with enough effort and careful design work it could help. See "Consider traits during dependency resolution" subsection of "Future directions":

The implementation to this proposal only considers traits after the dependency resolution when constructing the module graph. This is inline with how platform specific dependencies are currently handled. In the future, both platform specific dependencies and traits can be taken into consideration during dependency resolution to avoid fetching an optional dependency that is not enabled by a trait. Changing this doesn't require a Swift evolution proposal since it is just an implementation detail of how dependency resolution currently works.

@FranzBusch can clarify, since he's the primary author of the current implementation.

1 Like